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Introduction  to  the  Technology  Readiness  Assessment  (TRA)  Briefing 


A  TRA  is  a  formal,  metrics-based  process  that  is  conducted  to  evaluate  the  maturity  of  technologies  and  their 
individual  components  (termed  Critical  Technology  Elements  (CTEs)).  The  assessment  is  prepared  by  a  group  of 
independent  subject  matter  experts  (SMEs),  known  as  the  Independent  Review  Team  (IRT),  using  data  collected  by 
the  program  engineers  and  technical  staff.  The  metrics  used  are  the  Department  of  Defense  (DoD)  Technology 
Readiness  Levels  (TRLs)  for  either  hardware  or  software  systems. 

For  Major  Defense  Acquisition  Programs  (MDAPs)  to  proceed  into  Milestone  B,  a  TRL  6  (system/sub-system  model 
or  prototype  demonstration  in  a  relevant  environment)  is  required  for  all  CTEs — technologies  deemed  to  be  both  critical 
to  the  system’s  functionality  and  new  or  novel.  In  addition,  for  Milestone  B  MDAPs,  there  is  a  statutory  requirement  for 
certification  of  “demonstration  in  a  relevant  environment  by  the  Milestone  Decision  Authority  (MDA)”  (Title  10  U.S.C. 
§2366b).  Although  not  in  statute,  TRL  7  (system  prototype  demonstration  in  an  operational  environment)  is  an  exit 
criterion  for  the  Engineering  and  Manufacturing  Development  (EMD)  Phase  for  progress  into  Milestone  C. 

Therefore,  it  is  essential  (1)  that  assessments  are  prepared  for  and  performed  consistently  and  reliably  and  (2)  that 
all  team  members  are  familiar  with  the  rules  and  regulations  for  TRA  and  with  the  recommended  best  practices  for 
performing  the  assessment. 

The  TRA  briefing  included  in  this  document  is  designed  for  use  by  IRT  members  and  others  unfamiliar  with  the  TRA 
process.  The  briefing  should  help  all  persons  associated  with  the  evaluation  of  a  DoD  acquisition  program  to  become 
familiar  with  the  TRA  process,  regulations,  and  requirements.  A  thorough  understanding  of  the  required  process  early 
in  a  program’s  maturity  can  guide  the  program  toward  Milestone  decisions  at  the  appropriate  time.  The  goal  is  to 
reduce  cost  growth  and  schedule  slippages  that  occur  because  of  immature  technologies  that  might  enter  the  EMD 
Phase  of  the  Defense  Acquisition  System. 

This  briefing,  which  should  be  supplemented  by  the  Technology  Readiness  Assessment  (TRA)  Deskbook,  also 
provides  the  Director,  Research  Directorate  (DRD)  the  guidance  and  best  practices  for  conducting  TRAs.  Several 
examples  are  offered  to  assist  a  team  in  selecting  the  CTEs  and  the  expected  metrics  for  proper  evaluation. 
Procedures  should  be  based  upon  the  principles,  guidance,  and  recommended  best  practices  contained  in  these 
materials. 
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Executive  Summary: 
Main  Discussion  Points 


•  Related  legislation  and  policy 

•  Technology  Readiness  Assessment  (TRA)  Overview 

-  Definition  and  processes 

•  Succinct  program  definition  enables  determination  and 
evaluation  of  Critical  Technology  Elements  (CTEs) 

-  Use  of  Technology  Readiness  Levels  (TRLs) 

•  Hands-on  evaluation  of  CTEs  is  necessary  to  reach  TRL  6  or 
higher 

-  Demonstration  of  capability 

-  Relevant  environment 

•  Importance  and  outcomes 

•  Key  player  roles  and  responsibilities 


DoD  Technology  Maturation  Policy 
Leading  To  Milestone  B  Is  Unambiguous 


•  Technology  developed  in  science  and  technology  (S&T)  or 
procured  from  industry  or  other  sources  shall  have  been 
demonstrated  in  a  relevant  environment  or,  preferably,  in  an 
operational  environment  to  be  considered  mature  enough  to 
use  for  product  development 

•  Technology  readiness  assessments,  and  where  necessary, 
independent  assessments,  shall  be  conducted 

•  If  technology  is  not  mature,  the  DoD  Component  shall  use 
alternative  technology  that  is  mature  and  that  can  meet  the 
user's  needs 

(Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition 

System,  December  8,  2008,  Enclosure  2,  paragraph  5.d.(4)) 


and 
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The  Policy  Is  Reflected  as  a 
Statutory  Requirement  for  Certification 


Title  10  U.S.C.,  Subtitle  A,  Part  IV,  Chapter  139 

§2366b.  Major  defense  acquisition  programs: 
certification  required  before  Milestone  B 
or  Key  Decision  Point  B  approval 

(a)  Certification -  A  major  defense  acquisition 
program  may  not  receive  Milestone  B  approval, 
or  Key  Decision  Point  B  approval  in  the  case  of 
a  space  program,  until  the  milestone  decision 
authority- 

(2)  Further  certifies  that- 

the  technology  in  the  program  has  been  demon¬ 
strated  in  a  relevant  environment  [as  determined 
by  the  milestone  decision  authority  on  the  basis 
of  an  independent  review  and  assessment  by  the 
Director  of  Defense  Research  and  Engineering]. 
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Certification  submitted  with  the  first  Selected  Acquisition  Report  for  the  program 
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DoD  Policy  at  Milestone  C  for  Entry  Into 
Production  and  Deployment  Is  Also  Clear 


•  DoDI  5000.02  ,  Enclosure  2,  paragraph  7.b  Entrance 
Criteria.  Entrance  into  this  phase  depends  on  the 
following  criteria: 

-  Acceptable  performance  in  developmental  test  and 
evaluation  (DT&E)  and  operational  assessment 

-  Mature  software  capability 

-  No  significant  manufacturing  risks 

-  Acceptable  interoperability 

-  Acceptable  operational  supportability 


Technology  maturity  policy  does  not  distinguish 
Information  Technologies  from  technologies  in  general 


Technology  Readiness  Assessment  (TRA) 


•  A  systematic,  metrics-based  process  and  accompanying 
report  that 

-  Assesses  the  maturity  of  CTEs  used  in  systems 

-  Uses  TRLs  as  the  metric 

•  Adequate  performance  to  meet  program  requirements  must  be 
demonstrated  in  the  appropriate  environment 

-  Demonstrates 

•  How  the  CTEs  are  identified 

•  Why  CTEs  are  important  to  the  program 

•  An  independent  (from  the  program)  assessment  of  their 
maturity 


The  TRA  provides  feedback  to  the  program,  informs  milestone 
decisions,  and  supports  technology  certification  to  Congress 
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Program  manager  (PM)  responsibility 


Process  Overview 


PM  responsibility;  Coordinate  with 
S&T  Exec;  Keep  Director,  Research 
Directorate  (DRD)  informed 


Independent  Review  Team  (IRT) 
responsibility  in  conjunction  with 
program.  IRT  appointed  by  S&T  Exec 

Component  S&T  Exec  responsi¬ 
bility;  DRD  must  concur 


IRT  responsibility;  PM  funds  it  and 
provides  tech  support 


S&T  Exec  coordinates;  Acquisition 
Executive  submits 


DRD  responsibility 
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TRAs  Explicitly  Address 
Critical  Technology  Elements  (CTEs) 


•  A  technology  element  is  “critical” 

-  If  the  system  being  acquired  depends  on  this  technology 
element  to  meet  operational  requirements 

•  Within  acceptable  cost  and  schedule  limits 

and 

-  If  the  technology  element  or  its  application  is 

•  Either  new  or  novel,  or 

•  In  an  area  that  poses  major  technological  risk  during 
detailed  design  or  demonstration 


Assessment  focuses  on  the 
actual  technologies  from  the  program’s  design 
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Hardware  Technology  Readiness  Levels 

(TRLs)  6-7 


TRL 

Definition 

Description 

6 

System/subsystem  model  or  proto¬ 
type  demonstration  in  a  relevant 
environment. 

Representative  model  or  prototype 
system,  which  is  well  beyond  that  of 
TRL  5,  is  tested  in  a  relevant  environ¬ 
ment.  Represents  a  major  step  up  in 
a  technology’s  demonstrated  readi¬ 
ness.  Examples  include  testing  a 
prototype  in  a  high-fidelity  laboratory 
environment  or  in  a  simulated  opera¬ 
tional  environment. 

7 

System  prototype  demonstration  in 
an  operational  environment. 

Prototype  near  or  at  planned  opera¬ 
tional  system.  Represents  a  major 
step  up  from  TRL  6  by  requiring 
demonstration  of  an  actual  system 
prototype  in  an  operational  environ¬ 
ment  (e.g.,  in  an  aircraft,  in  a  vehicle, 
or  in  space). 
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Software  Technology  Readiness  Levels 

(TRLs)  6-7 


TRL 

Definition 

Description 

6 

Module  and/or  subsystem  validation 
in  a  relevant  end-to-end  environment. 

Level  at  which  the  engineering 
feasibility  of  a  software  technology  is 
demonstrated.  This  level  extends  to 
laboratory  prototype  imple-mentations 
on  full-scale  realistic  problems  in 
which  the  software  technology  is 
partially  integrated  with  existing 
hardware/software  systems. 

7 

System  proto-type  demonstration  in 
an  operational  high-fidelity 
environment. 

Level  at  which  the  program  feasibility 
of  a  software  technology  is  demon¬ 
strated.  This  level  extends  to  opera¬ 
tional  environment  prototype 
implementations,  where  critical  tech¬ 
nical  risk  functionality  is  available  for 
demonstration  and  a  test  in  which  the 
software  tech-nology  is  well  inte¬ 
grated  with  operational  hardware/ 
software  systems. 
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Understanding  Context  Is  Necessary 
To  Evaluate  Maturity  of  CTEs 


•  At  Milestone  B,  CTE  performance  must  be  demonstrated 
in  a  relevant  environment 

-  A  testing  environment  that  simulates  the  technologically 
stressing  aspects  of  the  operational  environment 

•  At  Milestone  C,  CTE  performance  must  be  demonstrated 
in  an  operational  environment 

-  An  environment  that  addresses  all  the  operational 
requirements  and  specifications  required  of  the  final 
system  to  include  platform/packaging 


Identification  of  CTEs  and  the  environment  requires  a  thorough 
understanding  of  system  requirements,  design,  and  architecture 
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All  Aspects  of  the 
Environment  Must  Be  Considered 


•  For  Information  Technology  (IT)-related  CTEs,  the 
environment  includes  physical,  logical,  data,  and 
security  environments 

-  Logical  environment  includes  other  applications,  run-time 
(operating  system,  middleware),  security  interfaces,  and 
Web  enablement 

-  Data  environment  includes  formats,  data  rates,  latency 

-  Security  environment  includes  firewalls,  appliques, 
methods  or  nature  of  attacks 


For  off-the-shelf  products,  the  environment  directly  impacts  the  TRL 
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Basis  of  Technology  Maturity 
Assessments  Throughout  Acquisition 


Milestone  A 

Milestone  B 

Milestone  C 

Basis  of  CTE 
Identification 

Early  evaluation  of 
technology  maturity 

Current  level  of 
design  and  Capa¬ 
bilities  Develop¬ 
ment  Document 
(CDD)  requirements 

Planned  LRIP 
article  (or  limited 
deployment  version 
of  an  IT  system), 
prior  TRAs,  and 
final  design 

CTE  Identification 
Status 

Potential  CTEs 

CTEs  -  actual 
technologies  in  a 
preliminary  design 

CTEs  of  planned 

LRIP  articles  (or 
limited  deployment 
version  of  an  IT 
system) 

Assessment 

Method 

Evaluated  in  early 
evaluations  of  tech¬ 
nology  maturity  and 
Technology  Matura¬ 
tion  Plans  (TMPs) 

Assessed  in 
Milestone  B  TRA 

Assessed  in 
Milestone  C  TRA 

Documentation 

Informal  submission 
to  DRD  and  corres¬ 
ponding  updates  to 
TDS  appendix 

Milestone  B  TRA 

Milestone  C  TRA 
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Outcomes  From  the  TRA  Process 


•  Programs  enter  Engineering  and  Manufacturing  Devel¬ 
opment  (EMD)  with  mature  technologies  and  avoid 
design  turbulence,  delay,  and  expense 

•  Oversight  authorities  certify  the  maturity  of  the  tech¬ 
nologies  with  confidence 

•  Systems  deploy  with  proven  technologies,  thereby 
delivering  known  behavior  and  avoiding  field  fixes 

•  Programs  identify  technologies  for  additional  matu¬ 
ration  and  later  insertion  into  the  system 


16 


PM  Roles  and  Responsibilities 


•  Plans  and  funds  the  program’s  risk  reduction  activities  to  ensure  that  CTEs 
reach  the  appropriate  maturity  levels 

•  Informs  the  Component  S&T  Executive  of  the  need  to  conduct  a  TRA 

•  Funds  the  TRA  evaluation  for  his  program 

•  Designates  a  responsible  individual  in  the  program  office  to  organize  all  TRA 
activities 

•  Prepares  a  draft  TRA  schedule  and  incorporates  the  approved  version  in  the 
program’s  IMP  and  IMS 

•  Suggests  to  the  Component  S&T  Executive  the  subject  matter  expertise 
needed  to  perform  the  TRA 

•  Ensures  that  the  IRT  is  familiar  with  the  program 

•  Identifies  possible  CTEs  for  IRT  consideration 

•  Provides  evidence  of  CTE  maturity  to  the  IRT  for  its  assessment,  including 
contractor  data 

•  Provides  technical  expertise  to  the  IRT  as  needed 

•  Drafts  the  section  of  the  TRA  report  containing  a  brief  description  of  the 
program  (program/system  overview,  objectives,  and  descriptions) 
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Component  S&T  Executive 
Roles  and  Responsibilities 


•  Directs  the  conduct  of  the  TRA 

•  Coordinates  on  the  TRA  schedule 

•  Nominates  SMEs  to  be  on  the  IRT 

•  Provides  the  DRD  the  credentials  of  ail  prospective  IRT  members 
and  sufficient  information  to  confirm  their  independence  from  the 
program 

•  Trains  IRT  members  on  the  TRA  process 

•  Reviews  the  TRA  report  and  prepares  the  TRA  report  cover  memo¬ 
randum,  which  may  include  additional  technical  information 
deemed  appropriate  to  support  or  disagree  with  IRT  findings 

•  Sends  the  completed  TRA  to  the  CAE  for  official  transmittal  to  the 
DRD  and  furnishes  an  advance  copy  to  the  DRD 

•  Maintains  continuity  in  the  IRT  membership  for  all  TRAs 
conducted  over  the  life  of  a  program,  to  the  maximum  extent 
possible 
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IRT  Roles  and  Responsibilities 


•  Keeps  the  Component  S&T  Executive  and  the  DRD  informed 
on  progress  throughout  the  entire  TRA  process 

•  Develops  a  list  of  CTE  candidates  in  conjunction  with  the  PM 

•  Assesses  the  TRLs  for  all  CTEs 

•  Prepares  (or  oversees  the  preparation  of)  elements  of  the 
TRA  report  including  (1)  the  IRT  credentials  and  (2)  IRT  delib¬ 
erations,  findings,  conclusions,  and  supporting  evidence 

-  The  assessment  process  should  not  be  constrained  to  a 
validation  of  a  “program-developed”  position  on  the  TRL 
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DRD  Roles  and  Responsibilities 


•  Concurs  with  the  TRA  schedule 

•  Concurs  with  the  composition  of  the  IRT 

•  Reviews  the  candidate  CTE  list  and  identifies  any  changes  necessary  to 
form  the  final  CTE  list.  Additions  to  the  list  can  include  any  special- 
interest  technologies  that  warrant  the  rigor  of  the  formal  TRA  process 

•  Exercises  oversight  by  monitoring  and  evaluating  the  TRA  process  and 
reviewing  the  TRA.  On  the  basis  of  that  review,  a  TRA  revision  may  be 
requested  or  the  DRD  may  conduct  its  own  Independent  Technical 
Assessment  (ITA) 

•  Sends  the  results  of  its  TRA  review  to  the  appropriate  Overarching  Inte¬ 
grated  Product  Team  (OIPT)  and/or  the  Defense  Acquisition  Board  (DAB) 

•  Provides  the  DDR&E  recommendations  concerning  certification 

•  Recommends  technology  maturity  language  for  an  Acquisition  Deci¬ 
sion  Memorandum  (ADM),  noting,  in  particular,  conditions  under  which 
the  new  technology  can  be  inserted  into  the  program 
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What  Is  a  TRA? 


•  Systematic,  metrics-based  process 
that  assesses  the  maturity  of  CTEs 

-  Uses  TRLs  as  the  metric 

•  Regulatory  information 
requirement  for  all  acquisition 
programs  at  Milestones  B  and  C 

-  Submitted  to  DRD  for  ACAT  ID  and 
1AM  programs,  including  space 
programs 


Not  a  risk  assessment 

Not  a  design  review 

Does  not  address 
system  integration 
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Critical  Technology  Element  (CTE)  Defined 


A  technology  element  is  “critical”  if  the  system  being 
acquired  depends  on  this  technology  element  to  meet 
operational  requirements  (within  acceptable  cost  and 
schedule  limits)  and  if  the  technology  element  or  its 
application  is  either  new  or  novel  or  in  an  area  that  poses 
major  technological  risk  during  detailed  design  or 
demonstration 


CTEs  may  be  hardware  or  software  at  the  subsystem  or  component  level 
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Why  Is  a  Milestone  B  TRA  Important? 


The  Milestone  Decision  Authority  (MDA)  uses  the 

information  to  support  a  decision  to  initiate  a  program 

-  Trying  to  apply  immature  technologies  has  led  to  technical, 
schedule,  and  cost  problems  during  systems  acquisition 

-  TRA  established  as  a  control  to  ensure  that  critical  technologies 
are  mature,  based  on  what  has  been  accomplished 

f  TRA  \ 

(  is  the  j 
V  basis!  J 

Congressional  interest  ^ 

-  MDA  must  certify  to  Congress  that  the  technology  in  programs 
has  been  demonstrated  in  a  relevant  environment  at  program 
initiation 


MDA  must  justify  any  waivers  for  national  security  to  Congress 
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Why  Is  a  Milestone  B  TRA  Important? 

(Continued) 


•  The  PM  uses  the  expertise  of  the  assessment  team  and  the 

rigor  and  discipline  of  the  process  to  allow  for 

-  Early,  in-depth  review  of  the  conceptual  product  baseline 

-  Periodic  in-depth  reviews  of  maturation  events  documented  as 
verification  criteria  in  an  associated  CTE  maturation  plan 

-  Highlighting  (and,  in  some  cases,  discovering)  critical 
technologies  and  other  potential  technology  risk  areas  that 
require  management  attention  (and  possibly  additional 
resources) 

•  The  PM,  Program  Executive  Office  (PEO),  and  CAE  use  the 

results  of  the  assessment  to 

-  Optimize  the  acquisition  strategy  and  thereby  increase  the 
probability  of  a  successful  outcome 

-  Determine  capabilities  to  be  developed  in  the  next  increment 

-  Focus  technology  investment 
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Why  Is  a  Milestone  B  TRA  Important? 

(Continued) 


•  For  IT  systems,  which  rely  heavily  on  off-the-shelf  com¬ 
ponents,  TRAs  have  increased  management’s  focus  on 
finding  CTEs  that  relate  specifically  to  IT  issues  (e.g., 
interfaces,  throughput,  scalability,  external  dependen¬ 
cies,  integration,  and  information  assurance) 

-  Since  many  IT  systems  have  experienced  problems  in 
these  areas,  the  TRA  has  proven  useful  in  understanding 
potential  problems  earlier  in  the  process,  when  solution 
options  are  easier  to  adopt  and  less  costly  to  implement 


These  red  boxes,  which  appear  on  Slides  26,  28,  52,  72,  and  77,  are  hyperlinked  to 
pages  toward  the  back  of  the  presentation  (under  the  “Hyperlinks”  Section:  see 
Slide  127).  Opening  the  hyperlink  will  take  you  to  the  page  in  question.  Once  on  that 
page,  you’ll  see  another  red  box  with  the  word  .  Opening  the  Return  hyperlink 

will  take  you  back  to  the  page  to  which  it  is  linked. 


IT  TRA 


Challenges 
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Why  Is  a  Milestone  C  TRA  Important? 


•  Reflects  the  resolution  of  any  technology  deficiencies 
that  arose  during  EMD 

•  Serves  as  a  check  that  all  CTEs  are  maturing  as 
planned,  especially  any  new  CTEs  identified  in  EMD 

•  Documents  successful  DT&E 

•  Confirms  expansion  of  performance  envelope  to 
“operational”  environment 

•  Avoids  technology-driven  operational  testing  problems 

-  Operational  testing  should  focus  on  “effective  and 
suitable” 
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Why  Is  a  Milestone  C  TRA  Important? 

(Continued) 


•  For  Major  Automated  Information  System  (MAIS)  programs  or 
software-intensive  systems  with  no  production  components: 

-  Examines  plans  for  maintenance  and  upgrades  to  ensure  that  no  new 
CTEs  are  involved 

-  Identifies  where  new  Milestone  Bs  are  needed  for  future  releases  to 
initiate  efforts  to  improve  performance  and  determines  the  architectural 
changes  necessary  to  support  these  future  releases 

-  Determines  whether  algorithms  will  transfer  successfully  when  host 
platforms  are  moved  and  full-scale  applications  are  initiated  in  a  real 
operational  environment 

-  Checks  technology  component  of  information  assurance  (IA)  before 
deployment 

-  Ensures  that  the  operational  environment  for  systems  to  deploy  has 
included  duress 


Software-Intensive 

Systems 


Outline 


*  Executive  Summary 

*  Introduction _ 

*  Technology  Maturation 

*  Identifying  Critical  Technology  Elements  (CTEs) 

-  CTE  Examples 

*  Assessing  CTE  Readiness 

-  CTE  Readiness  Examples 

*  The  TRA  Report 

*  Summary 

*  Hyperlinks 

*  Backup 
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Technology  Maturation  Policy 
Leading  to  Milestone  A 


the  lead  DoD  Component(s)  shall  prepare  an  AoA  [Analysis 
of  Alternatives]  study  plan  to  assess  preliminary  materiel 
solutions,  identify  key  technologies,  and  estimate  life-cycle 
costs.  The  purpose  of  the  AoA  is  to  assess  the  potential 
materiel  solutions  to  satisfy  the  capability  need  documented  in 
the  approved  ICD.” 

The  AoA  shall  assess  the  critical  technology  elements 
(CTEs)  associated  with  each  proposed  materiel  solution, 
including  technology  maturity,  integration  risk,  manufacturing 
feasibility,  and,  where  necessary,  technology  maturation  and 
demonstration  needs  ” 

(Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition 
System,  December  8,  2008,  Enclosure  2,  paragraphs  5.c.(5)  and  5.c.(6)) 
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Technology  Maturation  Policy  Leading  To 
Milestone  B  Is  Unambiguous 


“PMs  shall  reduce  technology  risk,  demonstrate  technologies 
in  a  relevant  environment,  and  identify  technology  alternatives 
prior  to  program  initiation.” 

(Department  of  Defense  Directive  (DoDD)  5000.01,  The  Defense  Acquisition  System, 

May  12,  2003,  Certified  current  as  of  November  20,  2007,  Enclosure  1,  paragraph  El. 1.14)) 


The  TRA  complements — but  does  not  diminish — the  PM’s  responsibility 
to  pursue  risk  reduction  efforts  prior  to  program  initiation  at  Milestone  B 
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Technology  Maturation  Policy  Leading  To 
Milestone  B  is  Unambiguous  (Continued) 


“The  project  shall  exit  the  Technology  Development  Phase  when 
an  affordable  program  or  increment  of  militarily  useful  capability 
has  been  identified;  the  technology  and  manufacturing  processes 
for  that  program  or  increment  have  been  assessed  and  demon¬ 
strated  in  a  relevant  environment;  manufacturing  risks  have  been 
identified;  a  system  or  increment  can  be  developed  for  production 
within  a  short  time  frame  (normally  less  than  5  years  for  weapon 
systems);  or  when  the  MDA  decides  to  terminate  the  effort.  ...  A 
Milestone  B  decision  follows  the  completion  of  Technology 
Development.” 

(Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition 
System,  December  8,  2008,  Enclosure  2,  paragraph  5.d.(7)) 
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Technology  Maturation  Policy  Leading  To 
Milestone  B  Is  Unambiguous  (Continued) 


“The  management  and  mitigation  of  technology  risk,  which  allows  less 
costly  and  less  time-consuming  systems  development,  are  crucial  parts 
of  overall  program  management  and  are  especially  relevant  to  meeting 
cost  and  schedule  goals. 

Objective  assessment  of  technology  maturity  and  risk  shall  be  a  routine 
aspect  of  DoD  acquisition.  Technology  developed  in  S&T  or  procured 
from  industry  or  other  sources  shall  have  been  demonstrated  in  a 
relevant  environment  or,  preferably,  in  an  operational  environment  to  be 
considered  mature  enough  to  use  for  product  development  (see  the 
Technology  Readiness  Assessment  (TRA)  Deskbook’  (Reference  (n)). 
Technology  readiness  assessments,  and  where  necessary,  independent 
assessments,  shall  be  conducted.  If  technology  is  not  mature,  the  DoD 
Component  shall  use  alternative  technology  that  is  mature  and  that  can 
meet  the  user’s  needs.” 

(Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition 
System,  December  8,  2008,  Enclosure  2,  paragraph  5.d.(4)) 
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(Ibid.,  Enclosure  2,  paragraph  5.c.(9)) 


Prototyping  and  Competition  Policy  Provides 
Technology  Maturation  Safeguards 


“Evolutionary  acquisition  requires  collaboration  among  the  user,  tester,  and 
developer.  . . .  Technology  development  preceding  initiation  of  an  increment 
shall  continue  until  the  required  level  of  maturity  is  achieved,  and  prototypes  of 
the  system  or  key  system  elements  are  produced,  and  a  preliminary  design  is 
completed.” 

“The  TDS  [Technology  Development  Strategy]  and  associated  funding  shall 
provide  for  two  or  more  competing  teams  producing  prototypes  of  the  system 
and/or  key  system  elements  prior  to,  or  through,  Milestone  B.  Prototype  systems 
. . .  shall  be  employed  to  reduce  technical  risk,  validate  designs  and  cost 
estimates,  evaluate  manufacturing  processes,  and  refine  requirements.” 

(Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition 
System,  December  8,  2008,  Enclosure  2,  paragraphs  2.b  and  5.c.(9)) 

•  Promotes  maturity  via 

-  More  rigorous  demonstrations  in  relevant  environments 

-  More  comprehensive  evidence  of  maturity 

-  Fewer  technical  problems  in  the  final  design 

-  Using  prototypes  for  accelerated  life-cycle  tests 

-  Providing  insight  into  production  issues 
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Request  for  Proposal  (RFP)  Policy 
Provides  Technology  Maturation  Safeguards 


“Final  RFPs  for  the  EMD  phase,  or  any  succeeding  acquisition 
phase,  shall  not  be  released,  nor  shall  any  action  be  taken  that 
would  commit  the  program  to  a  particular  contracting  strategy, 
until  the  MDA  has  approved  the  Acquisition  Strategy.  The  PM 
shall  include  language  in  the  RFP  advising  offerors  that  (1)  the 
government  will  not  award  a  contract  to  an  offeror  whose  pro¬ 
posal  is  based  on  CTEs  that  have  not  been  demonstrated  in  a 
relevant  environment  and  (2)  that  offerors  will  be  required  to 
specify  the  technology  readiness  level  of  the  CTEs  on  which 
their  proposal  is  based  and  to  provide  reports  documenting 
how  those  CTEs  have  been  demonstrated  in  a  relevant 
environment.” 

(Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition 
System,  December  8,  2008,  Enclosure  2,  paragraph  6.c.(4)) 


35 


Open  Dialogue  and  Feedback  on  AT&L  Policy 

(AT&L  Memo  Aug  24  2007) 


•  Policy 

-  “  .  .  .  structure  all  planned  competitions  with  one  or  more 
government  industry  feedback  and  dialogue  points  prior  to  receipt 
of  final  proposals.” 

-  “All  ongoing  competitions  should  be  reviewed  with  a  bias  toward 
incorporating  feedback  and  dialogue  sessions  before  receipt  of 
final  proposals.” 

•  Results  of  the  dialogue 

-  A  high-quality,  well-understood  proposal 

-  Allows  the  acquisition  team  to  explain  and  industry  to  understand 
the  fundamental  factors  that  determine  the  outcome  of  the 
competition 

-  Provides  multiple  inputs  for  the  government  to  define  the  required 
relevant  environment  for  candidate  CTEs  and  to  clarify  criteria 
with  contractors 
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The  Policy  Is  Reflected  as  a 
Statutory  Requirement  for  Certification 


Title  10  U.S.C.,  Subtitle  A,  Part  IV,  Chapter  139 

§2366b.  Major  defense  acquisition  programs: 
certification  required  before  Milestone  B 
or  Key  Decision  Point  B  approval 

(a)  Certification -  A  major  defense  acquisition 
program  may  not  receive  Milestone  B  approval, 
or  Key  Decision  Point  B  approval  in  the  case  of 
a  space  program,  until  the  milestone  decision 
authority- 

(2)  Further  certifies  that- 

the  technology  in  the  program  has  been  demon¬ 
strated  in  a  relevant  environment  [as  determined 
by  the  milestone  decision  authority  on  the  basis 
of  an  independent  review  and  assessment  by  the 
Director  of  Defense  Research  and  Engineering]. 


TM«  UCVITAMY  o*  ccn hU 

IC’O  sertHse 

Mt-Ni'C *.  0C  XNvKi 

ucrtisM 

trcm  \ll  0nTW»V7*r. 

KUC  rtUctm  *  T>*  11  te*  Cte 


«  AiteiOM  AiT  Ik  Ftu*  Vm  XCt  l  N*  c»  *Uimm 

m  Aateer*  MDJC »  Mum  Mw  Pnpm  (MZM^  *  ate 

.  <  |n«  »  MiMm  B  «  Em  B 


>CA  ■*»  «««*«■»«  mm*  CX}  if;  t(  *»  r*9JC*4 

at pMtinefcttl (to  «o*£»Mr  Ik  o 
>©A>  4  *-  VD  A  ******  te  bM  Ik  •  am  *■  W  teto 

WBM'OHUllMML  Mim  TV)  VIA  itel  MhK*t«RI« 

<i—nm  >rtm >* «< iW  w  Tb»  VZ>A km  w  611 

ttr<K  ttsSjscfy 


b  »*!aco  «  £»  ckt&kmc  smacsadja  a*  iC  A  tel  mkad*  «•  fcCme* 
nicar  cftiJCB 

n9S*l«nr»ii>»«  t.  Kites*!  mczec  214U  o£  z£s  S5.  ttsn!  Vm 

cm' 


5»yrA>.  Si*  oi  5o  VC  A>1  or-ifri  S 

9faM«fMm»CmmaAcpanEn(m<t«BiAnMSiaim 
•f  Mbm  Sbc  SV»«lt  of  btest&os  i 

Taue\«  >:■>  kJ  Lii_  U  an.-  rr.i  ox  ft 


Any  tenor 
At  tfK*4 


Certification  submitted  with  the  first  Selected  Acquisition  Report  for  the  program 
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. . .  and  for  Milestone  B  Certification  Changes 


(b)  Changes  to  Certification- 

(1)  The  program  manager  for  a  major  defense  acquisition  program 
that  has  received  certification  under  subsection  (a)  shall 
immediately  notify  the  milestone  decision  authority  of  any 
changes  to  the  program  that  - 

(A)  alter  the  substantive  basis  for  the  certification  of  the  MDA 
relating  to  any  of  the  components  of  such  certification;  or 

(B)  otherwise  cause  the  program  to  deviate  significantly  from 
the  material  provided  to  the  milestone  decision  authority  in 
support  of  such  certification. 

(2)  Upon  receipt  of  information  under  paragraph  (1),  the  milestone 
decision  authority  may  withdraw  the  certification  concerned  or 
rescind  Milestone  B  approval  (or  Key  decision  Point  B  approval 
in  the  case  of  a  space  program)  if  the  milestone  decision 
authority  determines  that  such  certification  or  approval  is  no 
longer  valid. 
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DoD  Practices  To  Support 
the  Statutory  Requirements 


•  Early  evaluations  of  technology  maturity  (prior  to  Milestone  A) 

are  necessary  to 

-  Provide  a  basis  for  modifying  the  requirements  if  technological 
risks  are  too  high 

-  Support  the  development  of  TMPs  that  show  how  all  likely  CTEs 
will  be  demonstrated  in  a  relevant  environment  before  preliminary 
design  begins  at  the  full  system  level 

-  Refine  the  TDS 

-  Inform  the  test  and  evaluation  (T&E)  community  about  technology 
maturity  needs 

-  Ensure  that  all  potential  CTEs  are  included  in  the  program’s  risk 
management  database  and  plan 

-  Articulate  external  dependencies  on  technology  base  projects 
and  define  specific  technologies,  technology  demonstration 
events,  and  exit  criteria  for  the  technology  to  transition  into  the 
acquisition  program 
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DoD  Practices  To  Support 
the  Statutory  Requirements  (Continued) 


•  USD(AT&L)  practice 

-  Programs  that  have  immature  technologies  will  not  be 
initiated  at  Milestone  B 

-  The  same  standards  apply  to  all  acquisition  programs 

•  As  directed  by  10  U.S.C.  2366b,  DDR&E  will  provide 
technical  advice  based  upon  an  independent  review 
and  assessment  to  the  MDA  in  support  of  certification 

-  For  MDAPs,  MAISs,  and  space  systems,  the  approved 
TRA  process,  as  found  in  the  DoD  TRA  Deskbook  report, 
will  be  the  basis  of  that  advice 

-  The  DDR&E-approved  TRA  process  takes  precedence 
over  other  guidance  in  situations  where  conflict  would 
arise,  pending  future  modification 
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TRA  Processes  Designed 
To  Support  This  Technical  Advice 


•  Safeguards  in  place  to  provide  the  DDR&E  the 
confidence  necessary  to  ensure  the  MDA  that 
certification  can  be  made 

-  To  ensure  that  the  TRA  supports  the  certification,  it  must 
draw  upon  the  best  technical  information  available 

•  As  such,  a  generic  TRA  not  based  on  the  planned  technical 
solution  is  not  acceptable 

•  The  TRA  must  be  based  on  the  technologies  in  the  system 

-  SMEs  must  identify  and  assess  the  CTEs 

•  These  experts  must  be  independent  of  the  program  (DDR&E 
concurrence  required) 

•  DDR&E  has  final  say  on  CTE  list 


41 


TRA  Processes  Designed  To  Support 
This  Technical  Advice  (Continued) 


•  Assurance  that  technologies  have  been  demonstrated  in  a 
relevant  environment  by  the  winning  EMD  Phase  contractor 

-  To  initiate  programs  with  mature  technologies,  the  source 
selection  process  should  include  a  focus  on  technical  maturity 

-  TRAs  must  be  performed  on  all  the  competitors  in  a  source 
selection 

•  ADM  language  establishing  conditions  for  CTE  insertion  after 
Milestone  B 

-  To  initiate  programs  with  mature  technologies,  immature  CTEs 
may  be  pursued  in  a  parallel  development  effort,  if  approved 
maturation  plans  submitted  with  the  TRA — on-ramp  vice  off-ramp 
for  preferred  approaches  with  undemonstrated  technologies 
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Outline 


*  Executive  Summary 

*  Introduction 

*  Technology  Maturation 

*  Identifying  Critical  Technology  Elements  (CTEs) 

-  CTE  Examples 

*  Assessing  CTE  Readiness 

-  CTE  Readiness  Examples 

*  The  TRA  Report 

*  Summary 

*  Hyperlinks 

*  Backup 


43 


Program  manager  (PM)  responsibility 


Process  Overview 


PM  responsibility;  Coordinate  with 
S&T  Exec;  Keep  Director,  Research 
Directorate  (DRD)  informed 


Independent  Review  Team  (IRT) 
responsibility  in  conjunction  with 
program.  IRT  appointed  by  S&T  Exec 

Component  S&T  Exec  responsi¬ 
bility;  DRD  must  concur 


IRT  responsibility;  PM  funds  it  and 
provides  tech  support 


S&T  Exec  coordinates;  Acquisition 
Executive  submits 


DRD  responsibility 
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Component  S&T  Executives 


•  Army 

-  Deputy  Assistant  Secretary  (Research  and  Technology) 

•  Navy 

-  Chief  of  Naval  Research  (CNR) 

•  Air  Force 

-  Deputy  Assistant  Secretary  (Science,  Technology,  and 
Engineering) 

•  Defense  Information  Systems  Agency  (DISA) 

-  Vice  Director 

•  Defense  Logistics  Agency  (DLA) 

-  Chief  Information  Officer  (CIO) 


Responsible  for  directing  the  TRA 
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Component 
SAT  Executive 
appoints.  PM 
funds 


Independent  Review 
Team  (IRT) 


•  Selected  from  pool  of 
recognized  experts 

-  DoD  Components 

-  Federally  Funded  Research 
and  Development  Centers 
(FFRDCs) 

-  Universities 

-  Government  agencies 

-  Industry 

-  National  Laboratories 


Technical  Work  Breakdown 
System  (WBS)  Elements 


Manufacturing 
Sensors  ' 

Missile  warning 
Communications 
r  Architecture 
Processing 
SurvivabHjSy 

jg^are^ 

InformatioTTjsystems 
Trzflmng 
Logistics 


R&M 

Crew  systems  . 
Antennas 
Structures 
Propulsion 

/  Electrical  system! 

“^Materials 
mt  Security 


Navigati 

Safety 


•  • 


V 


•  Final  team  membership  based  on  technical  WBS  where 
CTEs  are  located 


Responsible  for  performing  and  preparing  the  TRA 


Tests  for  IRT  Independence 


•  Members  should  be  sufficiently  independent  of  the 
developers  (government  or  industry)  as  not  to  be 
unduly  influenced  by  their  opinions  or  have  any  actual 
or  perceived  biases 

•  To  avoid  being  influenced  by  the  PM,  an  IRT  member 
should  not  be  directly  working  for  or  matrixed  to  the 
program  or  be  a  part  of  any  program  Integrated  Product 
Team  (IPT) 
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Program  Responsible  for 
Scheduling  and  Funding  the  TRA 


•  Establish/determine  contract  vehicle  for  funding 

-  CTE  identification 

-  Data  gathering 

-  IRT 

•  Training  and  preparation 

•  Assessments 

•  Report 

•  Travel 

-  Development  of  TMPs 

•  Integrate  TRA  plan  of  attack  and  milestones  into  the  IMS 
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IRT 

responsible  in 
conjunction  with 
the  PM 


Identifying 


Month  Month  Month  Month  Month  Month  Month  Month  Month  Month  Month  Month 
12  11  10  9  8  7  6  5  4  3  2  1 


Establish  TRA  Schedule 
Form  an  IRT 

Identify  Candidate  CTEs 
Finalize  CTEs  through  Coordination 
Collect  Evidence  of  Maturity 


Schedule  should  be  set  6  to  12  months  before  the 
Milestone  review,  depending  on  the  complexity  of  the  program 
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Management  Process 


CTE  Identification: 


•  Initial  review 

-  PM-led,  with  program  office,  system  contractors,  and 
government  labs 

-  Thorough,  disciplined,  and  conservative  approach 

-  Identifies  longer  list  of  possible  CTEs  to  ensure  that  no 
potential  CTE  is  overlooked 

-  Identifies  information  needed  to  determine  whether  the 
possible  CTEs  meet  the  criteria  in  the  definition 

•  Independent  review 

-  Conducted  by  team  of  experts  (i.e.,  the  IRT) 

-  Resolves  status  based  on  data  and  expertise 

-  Develops  candidate  CTE  list 
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IRT 
responsible  in 
conjunction  with 
the  PM 


CTE  Identification: 
Technical  Process 


Use  the  technical  WBS — or  system  or  software  architecture 
for  IT  systems— to  identify  CTE  candidates  by 

-  Establishing  the  functions  to  be  performed  by  each  system, 
subsystem,  or  component  throughout  the  technical  WBS 

-  Determining  how  the  functions  will  be  accomplished 


-  Identifying  the  technologies  needed  to  perform  those  functions  at 
the  desired  level 


•  Tech  Pubs  •  Construction/ 

•  Eng  Data  Conversion 

•  MgtData  •  Equipment 

•  Support  Data  Acquisition  or 

•  Data  Depository  Modernization 

•  Maintenance 


Initial  Spares  & 
Repair  Parts 


Peculiar 

Support  Equipment 
Common 

Support  Equipment 

Operational/ 
Site  Activation 


Airframe 
Propulsion 
AV  SYS  Software 
Comm/ID 
Central  Computer 
Fire  Control 
Auto  Flight  Control 
Weapons  Delivery 


DT&E 
OT&E 
Mock-ups 
T&E  Support 
Test  Facilities 


Equipment 

Services 

Facilities 


Adapted  from  MIL-HDBK-881 ,  Department  of  Defense  Handbook:  Work  Breakdown  Structure,  T1  January  1 998 
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IRT 

responsible  in 
conjunction  with _ 

thePAA^^echnical  Process  (Continued) 


CTE  Identification: 


s  •  Criticality  to  the  program  criteria 

|  ^  -  Does  the  technology  have  a  significant  impact  an 

2  ~  operational  requirement,  cost,  or  schedule? 

-  i 


See  Section  B.4  of  the  TRA  Deskbook  for  other  examples 


Aircraft  1 

Networked 

Communication 

System  Example 
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At  least  one  answer 
must  be  "yes" 


CTE  Identification: 


^-^Technical  Process  (Continued) 

Other  criteria 

-  Does  this  technology  pose  a  major  development  or 
demonstration  risk? 

-  Is  the  technology  new  or  novel? 

-  Has  the  technology  been  modified  from  prior  successful 
use? 

-  Has  the  technology  been  repackaged  such  that  a  new 
relevant  environment  is  applicable? 

-  Is  the  technology  expected  to  operate  in  an  environment 
and/or  achieve  a  performance  beyond  its  original  design 
intention  or  demonstrated  capability? 


Environment  key  to  “new  or  novel” 
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Examples  of  Technologies  Posing  a 
Major  Development  or  Demonstration  Risk 


•  The  intent  of  both  statute  and  policy  is  to  avoid  turbu¬ 
lence  during  EMD 

•  Technologies  that  are  not  new  or  novel  can  still  pose 
risk  of  turbulence 

•  An  expansive  interpretation  of  the  CTE  definition  will 
often  be  necessary  to  capture  such  technologies 

-  Radiation  hardening  has  been  a  repeated  source  of 
difficulty  during  the  development  of  satellite  systems 

-  Force  protection  will  attract  high-level  attention  throughout 
the  development  of  manned  combat  systems 
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Comp  SAT  Exec 
responsibility 


CTE  Identification: 
Coordination  Process 


•  DRD  reviews  the  candidate  CTE  list  developed  by  the 
IRT  and  identifies  any  changes  necessary  to  form  the 
final  CTE  list 

•  Additions  to  the  list  can  include  any  special-interest 
technologies  that  warrant  the  rigor  of  the  TRA  process 
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Environment  Examples 


•  Physical  Environment  -  For  instance,  mechanical  components, 
processors,  servers  and  electronics;  kinetic  and  kinematic;  thermal 
and  heat  transfer;  electrical  and  electromagnetic;  climatic/weather, 
temperature,  particulate;  network  infrastructure 

•  Logical  Environment  -  For  instance,  software  (algorithm) 
interfaces;  security  interfaces;  Web-enablement 

•  Data  Environment  -  For  instance,  data  formats  and  databases; 
anticipated  data  rates,  data  delay,  and  data  throughput;  and  data 
packaging  and  framing 

•  Security  Environment  -  For  instance,  connection  to  firewalls; 
security  appliques;  rates  and  methods  of  attack 

•  User  and  Use  Environment  -  For  instance,  scalability; 
upgradeability;  user  behavior  adjustments;  user  interfaces; 
organization  change/realignments  that  have  system  impacts; 
implementation  plan 


ijj. 

JL 


Others  may  be  relevant 
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Sample  Questions  To  Determine 
If  Environment  Is  New  or  Novel 


•  Is  the  physical/logical/data  environment  in  which  this  CTE 
has  been  demonstrated  similar  to  the  intended  environment? 
If  not,  how  is  it  different?  Is  the  difference  important? 

•  Is  the  CTE  going  to  be  operating  at  or  outside  the  usual 
performance  envelope?  Do  specifications  address  the 
behavior  of  the  CTE  under  these  conditions?  What  is  unique 
or  different  about  the  proposed  operations  environment? 

•  Do  test  data,  reports,  or  analysis  that  compare  the 
demonstrated  environment  to  the  intended  environment 
exist?  If  modeling  and  simulation  (M&S)  is  an  important 
aspect  of  that  comparison,  are  the  analysis  techniques 
common  and  generally  accepted? 


See  Section  B.3.2  of  the  TRA  Deskbook  for  more  questions 
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How  Many  CTEs  Should  Be  Identified? 


•  Do  not  miss  any 

-  System  performance,  program  schedule,  and  cost  could 
be  jeopardized 

•  Do  not  be  overly  conservative 

-  If  too  many  non-critical  technologies  are  treated  as  CTEs, 
energy  and  resources  may  be  diverted  from  the  few  tech¬ 
nologies  that  require  an  intensive  maturation  effort 


If  a  disciplined  process  leads  to  an  inordinate  number  of  CTEs, 
the  proposed  development  program  may  be  too  far-reaching 
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Data  Collection 


•  PM  collects  evidence  of  CTE  maturity 

-  Ongoing  process  throughout  CTE  identification 

-  May  include  component  and  subsystem  test  descriptions, 
analyses,  environments,  and  results 

-  Best  Practice:  evidence  should  be  as  objective  as  possible  and  align 
with  current  TMP's  documented  verification  criteria  for  achieving 
the  next  level 


Keep  DRD  informed.  May  suggest  additional  CTEs 
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CTEs  May  Not  Be  Glamorous 


Ship  Example 

•  A  highly  maneuverable,  load-carrying  vehicle  capable  of 
motion  in  any  direction  was  identified  as  a  CTE 

-  Intended  for  manual  and  autonomous  use 

•  Sensors  and  software  for  autonomous  travel  will  be  new, 
as  will  the  vehicle’s  use  within  the  sea  environment 

•  This  critical  technology  provides  significant  capability 
enhancement  over  existing  material  handling  equipment 
and  supports  the  reduced  manning  goal  of  the  ship 
program 
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CTEs  May  Not  Be  Associated 
With  a  Key  Performance  Parameter  (KPP) 


Ground  Vehicle  Example 

•  KPPs  concerned  interoperability  and  transportability  of  the 
vehicle  itself 

•  Operational  Requirements  Document  (ORD)  called  for  integra¬ 
tion  of  a  standoff  chemical  agent  detector 

-  The  mission-essential  function  is  to  detect  and  classify 

-  A  passive  infrared  (IR)  detection  system  that  detects  the  presence 
or  absence  of  chemical  warfare  agents  was  planned  for  the  vehicle 

-  The  detection  system  was  appropriately  identified  as  a  CTE 


Criticality  to  the  program  test  is  as  follows: 

Does  the  technology  directly  impact  an  operational  requirement ? 
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CTEs  May  Not  Be  Associated 
With  a  KPP  (Continued) 


Sensor  Example 

•  Two  technologies  were  inappropriately  excluded 

-  Hyperspectral  imagery:  New  technology.  Not  required  to 
meet  KPPs 

-  Aided  Target  Recognition  (ATR)  algorithms:  Used  to 
support  throughput  of  synthetic  aperture  radar  (SAR) 
imagery.  Not  required  to  meet  KPPs 


Enabling  technologies  should  not  be  excluded  from  being  CTEs 
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A  CTE  May  Be  in  Another  Program 


Ground  Vehicle  Example 

•  A  vehicle-mounted,  on-the-move,  chemical  agent  detector 
was  identified  as  a  CTE 

-  It  impacted  an  operational  requirement  and 

-  It  was  new 

•  The  proposed  solution  was  a  passive  IR  detection  system 
that  detects  the  presence  or  absence  of  chemical  warfare 
agents  and  was  an  independent  program  initiated  in  Sep¬ 
tember  1996  under  the  Joint  Program  Office  for  Chemical, 
Biological,  Radiological,  and  Nuclear  Defense 


The  term  of  art  is  “External  Dependency”: 

They  must  be  included  in  the  TRA  but  are  not  required  to  be  mature 
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Consider  All  Environments 


•  A  “tactical”  logistics  system  bought  commercial  off-the-shelf 
(COTS)  software  and  hardware  to  implement  inventory  con¬ 
trol  in  theater.  Prior  to  Milestone  B,  the  program  briefed  the 
IRT  on  the  intended  use  of  the  system:  in  large  logistics 
bases  and  theater  HQ  to  track  supplies  locally 

•  Based  on  the  program’s  brief,  the  IRT  found  one  CTE 

•  Just  prior  to  Milestone  B,  a  user  professed  the  need  to  use 
the  system  in  a  bandwidth-disadvantaged,  intermittent- 
connectivity,  high-latency  environment  where  ruggedization 
was  required.  This  need  was  not  inconsistent  with  the  term 
“tactical”  as  defined  in  the  CDD,  but  this  user’s  intent  was 
new  to  the  program  and  to  the  IRT 

•  The  Milestone  B  date  was  delayed  until  the  more  difficult 
definition  of  “tactical”  environment  could  be  established 
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When  to  Aggregate  CTEs 


•  A  communications  program  had  three  candidate  CTEs  in  the 
network  management  category  prior  to  Milestone  B 

-  CTE  (1)  was  a  software  module  that  diagnosed  network  health  by 
building  a  database  on  the  network  manager’s  control  station 

-  CTE  (2)  stored  information  on  those  links  that  were  able  send 
user  traffic 

-  CTE  (3)  stored  information  on  network  routing 

•  By  Program  Design  Review  (PDR),  no  data  were  to  be  stored 
on  the  network  manager’s  control  station  in  favor  of  a  distri¬ 
buted  solution.  Also,  the  information  on  user  traffic  and 
routing  was  to  be  collected  by  the  same  module  and  stored 
in  the  same  database 

•  At  Milestone  B,  DDR&E  agreed  with  the  IRT  decision  to 
remove  CTE  (1)  and  aggregate  CTE  (2)  and  CTE  (3)  into  a 
single  CTE  called  “Routing-Status” 
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When  to  De-aggregate  CTEs 


•  An  IT  program  had  to  automate  data  transfer  from  one  legacy 
system  to  another 

•  The  program  proposed  to  build  an  edge-device,  write  the  soft¬ 
ware  to  control  it,  and  integrate  it  with  legacy  systems.  At  Mile¬ 
stone  B,  the  IRT  identified  the  edge-device  as  a  CTE  and 
assessed  it  as  TRL  6 

•  Before  Milestone  C,  the  IRT  deadlocked  on  whether  the  edge 
device  was  TRL  7.  The  device  was  like  a  laptop  (i.e.,  it  plugged 
into  the  interfaces  and  the  software  ran  on  it),  but  all  the  soft¬ 
ware  functionality  had  not  been  tested  with  all  legacy  systems 

•  The  solution  was  to  break  the  CTE  into  a  hardware  CTE  (TRL  7) 
and  a  software  CTE  (TRL  6).  More  testing  was  done  on  the 
software 
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Assessing 
CTE  Readiness 


Month  Month  Month  Month  Month  Month  Month  Month  Month  Month  Month  Month 


12  11  10  9 

Assess  CTE  Maturity 

Prepare,  Coordinate,  Submit  TRA  Report 

DRD  Review  &  Evaluation 

Perform  Independent  TRA  (if  necessary) 
Prepare  Evaluation  Memo 
Milestone  Review 


8  7  6  5  4  3  2  1 
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TRL  Overview 


•  Measures  technology  maturity 

•  Indicates  what  has  been  accomplished  in  the  develop¬ 
ment  of  a  technology 

-  Theory,  laboratory,  field 

-  Relevant  environment,  operational  environment 

-  Subscale,  full  scale 

-  Breadboard,  brassboard,  prototype 

-  Reduced  performance,  full  performance 

•  Does  not  indicate  that  the  technology  is  right  for  the 
job,  that  application  of  the  technology  will  result  in 
successful  development  of  the  system,  or  how  difficult 
the  application  might  be  to  implement 
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Increasing  maturity 


Hardware  TRLs:  Assessment  Criteria 


1.  Basic  principles  observed  and  reported 

2.  Technology  concept  and/or  application  formulated 

3.  Analytical  and  experimental  critical  function  and/or  charac¬ 
teristic  proof  of  concept 

4.  Component  and/or  breadboard  validation  in  a  laboratory 
environment 

5.  Component  and/or  breadboard  validation  in  a  relevant 
environment 


System/subsystem  model  or  prototype  demonstration  in  a 
relevant  environment 

System  prototype  demonstration  in  an  operational 
environment 

Actual  system  completed  and  qualified  through  test  and 
demonstration 

Actual  system  proven  through  successful  mission  operations 


See  additional  hardware  examples  in  Section  C.2  of  the  TRA  Deskbook 


Hardware  CTE 
Example 
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TRL  4  Hardware 

Minimum  Maturity  at  Milestone  A 


•  Definition:  Component  and/or  breadboard  validation  in  a 
laboratory  environment 

•  Description:  Basic  technological  components  are  integrated 
to  establish  that  they  will  work  together.  This  is  relatively 
“low  fidelity”  compared  with  the  eventual  system.  Examples 
include  integration  of  “ad  hoc”  hardware  in  the  laboratory 

•  Supporting  Information:  System  concepts  that  have  been 
considered  and  results  from  testing  laboratory-scale  bread¬ 
board^).  References  to  who  did  this  work  and  when.  Pro¬ 
vides  an  estimate  of  how  breadboard  hardware  and  test 
results  differ  from  the  expected  system  goals 
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TRL  5  Hardware 


•  Definition:  Component  and/or  breadboard  validation  in  a 
relevant  environment 

•  Description:  Fidelity  of  breadboard  technology  increases 
significantly.  The  basic  technological  components  are  inte¬ 
grated  with  reasonably  realistic  supporting  elements  so  they 
can  be  tested  in  a  simulated  environment.  Examples  include 
“high-fidelity”  laboratory  integration  of  components 

•  Supporting  Information:  Results  from  testing  a  laboratory 
breadboard  system  are  integrated  with  other  supporting 
elements  in  a  simulated  operational  environment.  How  does 
the  “relevant  environment”  differ  from  the  expected  opera¬ 
tional  environment?  How  do  the  test  results  compare  with 
expectations?  What  problems,  if  any,  were  encountered? 
Was  the  breadboard  system  refined  to  more  nearly  match 
the  expected  system  goals? 
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TRL  6  Hardware 

Minimum  Maturity  at  Milestone  B 


•  Definition:  System/subsystem  model  or  prototype  demon¬ 
stration  in  a  relevant  environment 

•  Description:  Representative  model  or  prototype  system,  which 
is  well  beyond  that  of  TRL  5,  is  tested  in  a  relevant  environ¬ 
ment.  Represents  a  major  step  up  in  a  technology’s  demon¬ 
strated  readiness.  Examples  include  testing  a  prototype  in  a 
high-fidelity  laboratory  environment  or  in  a  simulated  opera¬ 
tional  environment 

•  Supporting  Information:  Results  from  laboratory  testing  of  a 
prototype  system  that  is  near  the  desired  configuration  in 
terms  of  performance,  weight,  and  volume.  How  did  the  test 
environment  differ  from  the  operational  environment?  Who 
performed  the  tests?  How  did  the  test  compare  with  expecta¬ 
tions?  What  problems,  if  any,  were  encountered?  What  are/ 
were  the  plans,  options,  or  actions  to  resolve  problems  before 
moving  to  the  next  level? 
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TRL  7  Hardware 

Minimum  Maturity  at  Milestone  C 


•  Definition:  System  prototype  demonstration  in  an  operational 
environment 

•  Description:  Prototype  near  or  at  planned  operational  system. 
Represents  a  major  step  up  from  TRL  6  by  requiring  demon¬ 
stration  of  an  actual  system  prototype  in  an  operational 
environment  (e.g.,  in  an  aircraft,  in  a  vehicle,  or  in  space). 
Examples  include  testing  the  prototype  in  a  test  bed  aircraft 

•  Supporting  Information:  Results  from  testing  a  prototype 
system  in  an  operational  environment.  Who  performed  the 
tests?  How  did  the  test  compare  with  expectations?  What 
problems,  if  any,  were  encountered?  What  are/were  the  plans, 
options,  or  actions  to  resolve  problems  before  moving  to  the 
next  level? 
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Increasing  maturity 


Software  TRLs:  Assessment  Criteria 


1. 

2. 

3. 

4. 


5. 

6. 


Basic  principles  observed  and  reported 

Technology  concept  and/or  application  formulated 

Analytical  and  experimental  critical  function  and/or  charac¬ 
teristic  proof  of  concept 

Module  and/or  subsystem  validation  in  a  laboratory  environment 
(i.e.,  software  prototype  development  environment) 

Module  and/or  subsystem  validation  in  a  relevant  environment 

Module  and/or  subsystem  validation  in  a  relevant  end-to-end 
environment 

System  prototype  demonstration  in  an  operational  high-fidelity 
environment 

Actual  system  completed  and  mission  qualified  through  test  and 
demonstration  in  an  operational  environment 

Actual  system  proven  through  successful  mission  proven 
operational  capabilities 


Software  CTE 

See  additional  software  examples  in  Section  C.3  of  the  TRA  Deskbook 

Example 
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TRL  4  Software 

Minimum  Maturity  at  Milestone  A 


•  Definition:  Module  and/or  subsystem  validation  in  a  labora¬ 
tory  environment  (i.e.,  software  prototype  development 
environment) 

•  Description:  Basic  software  components  are  integrated  to 
establish  that  they  will  work  together.  Their  efficiency  and 
robustness  are  relatively  primitive  compared  with  the  even¬ 
tual  system.  Architecture  development  initiated  to  include 
interoperability,  reliability,  maintainability,  extensibility, 
scalability,  and  security  issues.  Emulation  with  current/ 
legacy  elements  as  appropriate.  Prototypes  developed  to 
demonstrate  different  aspects  of  eventual  system 

•  Supporting  Information:  Advanced  technology  development, 
stand-alone  prototype  that  solves  a  synthetic  full-scale 
problem  or  a  stand-alone  prototype  that  processes  fully 
representative  data  sets 
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TRL  5  Software 


•  Definition:  Module  and/or  subsystem  validation  in  a  relevant 
environment 

•  Description:  Level  at  which  software  technology  is  ready  to 
start  integration  with  existing  systems.  The  prototype  imple¬ 
mentations  conform  to  target  environment/interfaces.  Experi¬ 
ments  with  realistic  problems.  Simulated  interfaces  to  existing 
systems.  System  software  architecture  established.  Algo¬ 
rithms  run  on  a  processor(s)  that  has  the  characteristics 
expected  in  the  operational  environment 

•  Supporting  Information:  System  architecture  diagram  around 
technology  element  with  critical  performance  requirements 
defined.  Processor  selection  analysis,  Simulation/Stimulation 
(Sim/Stim)  Laboratory  buildup  plan.  Software  placed  under 
configuration  management.  COTS/GOTS  (government  off-the- 
shelf)  components  in  the  system  software  architecture  are 
identified 
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TRL  6  Software 

Minimum  Maturity  at  Milestone  B 


•  Definition:  Module  and/or  subsystem  validation  in  a  relevant 
end-to-end  environment 

•  Description:  Level  at  which  the  engineering  feasibility  of  a 
software  technology  is  demonstrated.  This  level  extends  to 
laboratory  prototype  implementations  on  full-scale  realistic 
problems  in  which  the  software  technology  is  partially  inte¬ 
grated  with  existing  hardware/software  systems 

•  Supporting  Information:  Results  from  laboratory  testing  of  a 
prototype  package  that  is  near  the  desired  configuration  in 
terms  of  performance,  including  physical,  logical,  data,  and 
security  interfaces.  Comparisons  between  tested  environ¬ 
ment  and  operational  environment  analytically  understood. 
Analysis  and  test  measurements  quantifying  contribution  to 
system-wide  requirements  such  as  throughput,  scalability, 
and  reliability.  Analysis  of  human-computer  (user  environment) 
begun 
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TRL  7  Software 

Minimum  Maturity  at  Milestone  C 


•  Definition:  System  prototype  demonstration  in  an  opera¬ 
tional,  high-fidelity  environment 

•  Description:  Level  at  which  the  program  feasibility  of  a  soft¬ 
ware  technology  is  demonstrated.  This  level  extends  to  the 
operational  environment  prototype  implementations,  where 
critical  technical  risk  functionality  is  available  for  demon¬ 
stration  and  a  test  is  available  in  which  the  software  tech¬ 
nology  is  well  integrated  with  operational  hardware/software 
systems 

•  Supporting  Information:  Critical  technological  properties  are 
measured  against  requirements  in  a  simulated  operational 
environment 
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TRL  6  and  TRL  7  Comparison 


TRL  6 

TRL  7 

By  when  should  a  CTE 
be  at  least ...? 

Milestone  B 

Milestone  C 

What  Is  being 
assessed? 

CTE  as  part  of  a 
system/subsystem 
model  or  prototype 

CTE  as  part  of  a 
system  prototype 

What  is  the  assess¬ 
ment  environment? 

Relevant  environment 

Operational  environment 

Sample  environment 

High-fidelity  lab  or 
simulated,  operational 
environment 

Test  bed  or 
test  range  facility 
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Milestone  B  Requirement:  Demonstration  or 
Validation  in  a  Relevant  Environment  (TRL  6) 


Relevant  Environment:  a  set  of  stressing  conditions  represen¬ 
tative  of  the  full  spectrum  of  relevant  operational  employments, 
which  are  applied  to  a  CTE  as  part  of  a  component  (TRL  5)  or 
system/subsystem  (TRL  6)  model  or  prototype  to  identify  whether 
any  design  changes  or  fixes  are  needed  to  support  the  required 
(threshold)  functionality 
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Milestone  C  Requirement:  Demonstration  or 
Validation  in  an  Operational  Environment  (TRL  7  or  8) 


Operational  Environment:  a  set  of  operational  conditions , 
representative  of  the  full  spectrum  of  operational  employments, 
which  are  applied  to  a  CTE  as  part  of  a  system  prototype  (TRL  7) 
or  actual  system  (TRL  8)  to  identify  whether  any  previously 
unknown  or  undiscovered  design  problems  will  impact  the 
required  (threshold)  functionality 
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Demonstration  or  Validation  of  a  Technology 
in  an  Operational  Environment 


•  Requires  successful  trial  testing  that  either 

-  Shows  that  the  technology  satisfies  functional  need 
across  the  full  spectrum  of  operational  employments  or 

-  Shows  that  the  technology  satisfies  the  functional  need 
for  some  important  operational  employment  and  uses 
accepted  techniques  to  extend  confidence  over  all 
required  operational  employments 
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Assessing  COTS  Hardware 
The  Situation 


•  TRA  on  an  upgrade  to  a  major  DoD  aircraft 

•  CTEs  identified  and  grouped  based  on  WBS  categories 

•  Upgrade  centered  on  use  of  commercial  engines  and 
pylons  that  have  had  extensive  commercial  usage 

-  Pylons,  wing  attachments,  and  thrust  reversers  were  used 
commercially  in  similar  application 

-  Military  application  included  use  of  thrust  reversers  in  flight 

-  Most  algorithms  in  engine  software  unchanged 

•  Initial  assessments  of  TRL  8  or  9  for  propulsion 
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Assessing  COTS  Hardware  (Continued) 

What  Happened 


•  Discoveries  after  CDR 

-  Pylon  and  wing  attachment  not  strong  enough  for  asym¬ 
metric  thrust  reversal  in  flight 

-  Redesign  #1 :  Put  cables  across  bottom  of  engine 

•  Maintenance  burden  and  risk  of  mispositioning  the  thrust 
reversers 

-  Redesign  #2:  Connect  thrust  reverser  sections  across  top 
of  engine 

•  Structure  of  pylon  changed  slightly  as  well 
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Assessing  COTS  Hardware  (Continued) 

Lessons  Learned 


•  Expansive  definitions  of  the  “relevant”  or  “operational” 
environment  will  forestall  problems 

•  COTS  equipment,  when  adopted,  must  have  its  usage 
base  rigorously  compared  against  operational  environ¬ 
ments  when  identifying  or  assessing  CTEs 


Turbulence  during  design  can  result  from  any 
undemonstrated  aspect  of  the  technology/environment  combination 
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Assessing  COTS  Software 
The  Situation 


•  An  identity  management  program  was  assessed  by  an 

IRT  prior  to  Milestone  C 

-  The  IRT  conducted  an  industry  survey  and  assessed 
some  small  DoD  pilot  programs 

-  The  technology  scaled  to  the  DoD  size,  and  the  same 
commercial  sector  functions  were  to  be  used 

-  The  IRT  determined  that  all  the  CTEs  were  TRL  9  based  on 
their  commercial  use 
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Assessing  COTS  Software  (Continued) 

What  Happened 


•  DDR&E  agreed  with  the  IRT’s  conclusions  on  seal- 
ability;  however,  DDR&E  noted  that  the  security  envi¬ 
ronment  was  different 

-  In  this  case,  important  security  aspects  of  the  operational 
environment  were  overlooked.  The  DoD  faces  threats  that 
private  entities  do  not  face,  and  it  has  a  unique  risk  toler¬ 
ance  (it  must  self-ensure  at  the  cost  of  life  and  death) 

-  When  the  CTEs  were  reassessed  (including  the  DoD 
security  environment),  the  IRT  concluded  that  these  CTEs 
were  TRL  6.  They  were  demonstrated  to  provide  identity 
management  capability — but  only  in  a  benign  environment 
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Assessing  COTS  Software  (Continued) 

Lessons  Learned 


•  The  IRT  did  a  good  job  on  the  assessment  but  used 
incorrect  standards  for  a  successful  demonstration  in 
an  operational  environment 

-  The  operational  environment  must  include  the  full  spec¬ 
trum  of  stressing  events  that  can  be  expected  in  DoD  use 

-  Ostensibly  subtle  differences  in  the  way  COTS  software  is 
used  can  actually  lead  to  great  technical  risk  with  DoD  use 
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Getting  the  Right  Data 
The  Situation 


•  A  communications  program  was  approaching  Milestone  B 

-  The  program  scheduled  system-wide  test  events  that  assessed 
overall  system  capability  and  could  be  measured  against  KPPs 

-  The  CTEs  were  a  disaggregated  set  of  technologies  that  sup¬ 
ported  the  KPPs  and  other  requirements 

-  When  the  test  reports  were  sent  to  the  IRT,  not  enough  data  were 
available  to  determine  if  each  CTE  had  performed  correctly.  For 
example,  a  CTE  related  to  monitoring  communications  links  was 
supposed  to  demonstrate  accuracy  and  timeliness.  The  percent¬ 
age  of  messages  transmitted  by  the  all  the  links  was  in  line  with 
the  KPPs,  but  the  IRT  determined  that  the  TRL  of  the  monitoring 
CTE  could  not  be  determined  because  no  one  had  kept  track  of 
how  long  updates  took  or  whether  these  updates  were  consistent 
with  the  ground  truth 


94 


Getting  the  Right  Data  (Continued) 

What  Happened 


•  The  program  was  required  to  retest  the  monitoring  CTE 
and  the  other  CTEs,  at  great  monetary  cost 

-  After  retesting,  the  monitoring  CTE  was  found  to  be  TRL  6 

-  Another  CTE  was  found  not  to  be  TRL  6  because  in  certain 
conditions  that  were  likely  in  battle  but  occurred  a  just  few 
times  during  the  system-wide  tests,  the  CTE  failed  to  per¬ 
form  consistently 

-  The  program  and  contractor  installed  a  fix.  The  failed  CTE 
was  rated  TRL  6  four  months  later 
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Getting  the  Right  Data  (Continued) 

Lessons  Learned 


•  The  maturity  assessments  used  by  DDR&E  to  establish 
CTE  maturity  often  need  more  information  than  measures 
of  system-wide  capability 

•  TRLs  are  established  by  meeting  objective  standards 
explicitly  stated,  or  derived  from,  KPPs,  other  require¬ 
ments,  policy,  regulations,  and  even  law 
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A  Good  Logic  Chain 
The  Situation 


•  A  classified  IT  program  was  approaching  Milestone  B 

-  The  program  tried  to  schedule  tests  that  included  full-scale, 
realistic  problems,  but  the  contractor  had  fallen  behind  on 
the  delivery  of  key  items 

-  The  IRT  felt  that  TRL  6  could  not  be  established  without 
demonstrating  the  CTEs  in  these  full-scale  realistic 
problems 

-  Further,  the  requirements  for  the  some  CTEs  were  so 
general  that  the  IRT  did  not  know  how  to  make  a 
quantitative  objective  assessment 
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A  Good  Logic  Chain  (Continued) 

What  Happened 


•  DDR&E  agreed  with  the  IRT  but  pointed  out  that  technologies  similar 
to  the  CTEs  may  have  been  demonstrated  in  other  programs.  Also, 
DDR&E  said  that  the  IRT  should  develop  an  expert  position  on  the 
quantitative  standards  for  CTEs 

-  The  IRT  and  the  Program  Management  Office  (PMO)  cooperated  to  search 
external  records  and  reports  that  were  related  to  the  technology  in  the 
program 

-  The  IRT  found  that  similar  technology  had  been  demonstrated  in  a 
slightly  different  context.  The  TRA  included  a  compelling  chain  of  logic 
that  indicated  that  these  external  tests  were  sufficient  to  establish  TRL  6 
for  the  program’s  CTEs.  The  chain  of  logic  described  the  similarities  and 
differences  in  the  intended  use,  the  test  events,  and  the  metrics  for 
successful  demonstrations 

-  To  solve  the  ambiguous  requirements  problem,  the  IRT  used  law,  DoD- 
wide  policy,  the  organizational  regulations,  and  even  operational  data  to 
determine  what  quantitative  measures  and  standards  matched  the  quali¬ 
tative  words  in  the  programmatic  requirements  documents 
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A  Good  Logic  Chain  (Continued) 
Lessons  Learned 


•  By  presenting  complete,  clear,  compelling,  and  factual 
arguments,  the  TRA  put  together  disparate  pieces  of 
evidence  to  establish  TRL  6  for  all  CTEs 

•  When  faced  with  a  set  of  ambiguous  or  subjective 
requirements,  additional  sources  (to  those  of  the 
program’s  requirements  documents)  can  be  used  to 
clarify  what  a  constitutes  a  successful  demonstration 
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Scalability 
The  Situation 


•  A  communications  networking  program  was  approaching 
Milestone  B 

-  The  program  scheduled  full-scale,  realistic  tests 

-  The  IRT  analysis  compared  physics-based  M&S  predictions 
to  measured  performance 

-  The  results  showed  that  antenna  and  network  “self-healing” 
CTEs  were  performing  in  the  field  as  predicted.  The  results 
also  showed  that  the  CTEs  related  to  autonomous  network 
management  were  not  performing  as  predicted.  Further, 
CTEs  related  to  video  conferencing  were  trending  as  pre¬ 
dicted  but  were  not  meeting  the  standards  for  a  successful 
demonstration 
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Scalability  (Continued) 
What  Happened 


•  The  DDR&E  memo  to  the  MDA  outlined  the  situation.  The  MDA 

restructured  the  program 

-  The  antennas  and  self-healing  CTEs  were  incorporated  into  a  new 
increment.  They  progressed  rapidly  to  TRL  7  and  were  fielded  to 
the  warfighter 

-  The  autonomous  network  management  CTEs  continued  technol¬ 
ogy  development  and  ultimately  revealed  problems  in  both  the 
CTEs  and  the  models.  They  have  since  passed  Milestone  B,  and 
the  improved  models  are  in  wide  use 

-  The  video  conferencing  CTEs  were  shown  to  have  reached  funda¬ 
mental  scaling  limits  given  that  were  not  captured  in  the  modeling. 
The  original  predictions  of  performance  were  based  on  assump¬ 
tions  about  commercial  networks  with  commercial  traffic  profiles. 
The  technology  is  currently  back  in  the  tech  base 
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Scalability  (Continued) 
Lessons  Learned 


•  Scalability  must  be  demonstrated  at  Milestone  B.  M&S 
can  contribute  to  the  demonstration,  but  the  models 
must  be  reliable  and  predictive  for  the  relevant 
environment 

•  When  a  program  has  mature  and  immature  technology 
heading  into  Milestone  B,  the  TRA  can  assist  in  getting 
the  technology  that  is  “ready”  to  the  warfighter  sooner 
and  refocus  tech  development  efforts  on  the  immature 
parts 
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Information  Assurance  (IA) 
The  Situation 


•  An  intrusion  detection  system  was  approaching 
Milestone  C 

-  The  program  had  scheduled  system-wide  integration 
testing  in  a  controlled  environment  but  had  not  scheduled 
adversarial  testing 

-  The  IRT  analysis  assessed  no  greater  than  TRL  6  because 
the  environments  included  in  testing  were  not  operational 
environments.  As  components  or  subsystems,  the  CTEs 
had  been  tested  under  duress  for  TRL  6,  but  they  had  not 
been  tested  as  they  would  have  to  operate  in  a  fully  inte¬ 
grated  system.  The  testing  did  not  include  the  kind  of 
threats  the  system  should  expect  to  face 
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IA  (Continued) 
What  Happened 


•  DDR&E  agreed  with  the  IRT.  The  CTEs  had  been  demon¬ 
strated  under  duress  individually  but  not  as  a  system  in 
an  operational  environment 

-  A  Red-Team  effort  was  started  to  challenge  the  system 

-  The  results  were  important  and  informative 
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IA  (Continued) 
Lessons  Learned 


•  Information  systems  often  enter  initial  operational  test  and 
evaluation  (IOT&E)  in  actual-use  pilots.  Undetected  flaws 
in  the  system  can  allow  adversaries  access  to  the  DoD 
network 

•  Since  implementation  is  a  critical  part  of  security  and 
the  programs  that  comprise  the  Global  Information  Grid 
(GIG)  have  many  interdependencies,  adversarial  testing 
to  assess  lA-related  CTEs  in  a  cyber  warfare  environ¬ 
ment  is  critical  at  Milestone  C 
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Maturity  Demonstrated 
by  Others  May  Not  Suffice 


•  Early  in  the  Javelin  development,  the  prime  contractor 
produced  an  effective  system;  however,  the  focal  planes 
could  not  be  produced  by  the  prime  in  sufficient 
quantity  or  with  sufficient  yield 

-  A  different  IR  contractor  did  have  the  ability  and  was 
brought  on  as  a  subcontractor/supplier  of  the  focal  planes 

•  To  establish  maturity,  it  is  not  sufficient  for  “someone” 
to  be  able  to  demonstrate  the  needed  performance.  The 
technology  performance  must  be  deliverable  by  the 
performing  team 
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Non-Complex  Technology  Is  Unacceptable 
Rationale  for  TRL  6  or  Higher 


Example 

•  Subsystem  identified  as  CTE 

-  A  similar  or  existing  prototype  has  not  demonstrated  an 
ability  to  perform  the  subsystem’s  mission  from  the  example 
platform 

•  Inappropriately  assessed  at  TRL  6 

-  Although  the  subsystem  is  in  concept  design,  its  low  tech¬ 
nical  complexity  will  allow  the  use  of  known  and  proven 
fabrication  methods  and  materials 


Demonstration  of  a  prototype  in  a 
relevant  environment  is  precondition  for  TRL  6 
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System  Level  Demonstration 
Required  for  TRL  7  or  Higher 


Software-Intensive  System  Example 

•  TRA  inappropriately  identified  two  CTEs  as  TRL  7  and  two  as 
TRL  8 

•  The  rationale  for  the  TRL  scores  was  that  the  systems  being 
scored  are  currently  in  operational  use  and  have  already  been 
through  the  acquisition  process.  Integration  into  a  common 
environment  is  the  major  area  to  be  addressed  for  each  crit¬ 
ical  technology. 

•  The  IRT  approached  the  integration  issue  from  the  standpoint 
that  integration  will  occur  during  EMD,  and,  therefore,  the  TRL 
score  is  based  on  the  individual  critical  technology 

•  CTEs  should  have  been  assessed  to  be  TRL  6 
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Outline 


*  Executive  Summary 

*  Introduction 

*  Technology  Maturation 

*  Identifying  Critical  Technology  Elements  (CTEs) 

-  CTE  Examples 

*  Assessing  CTE  Readiness 

-  CTE  Readiness  Examples 

*  The  TRA  Report 

*  Summary 

*  Hyperlinks 

*  Backup 


TRA  Report 


•  Technical  report 

-  Short  description  of  the  program 

-  IRT  credentials 

-  IRT  deliberations,  findings,  conclusions,  supporting 
evidence,  and  major  dissenting  opinions 

-  Other  technical  information  deemed  pertinent  by  the 
Component  S&T  Executive 

•  Cover  letter  signed  by  the  Component  S&T  Executive 
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IRT  oversees 

preparation^-  ^^Technical  Report  Contents 
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1.0  Purpose  of  This  Document 
2.0  Program  Overview 

2.1  Program  Objective  0  o  O 

2.2  Program  Description,  Including 
Spirals  Covered 

2.3  System  Description 
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3.0  Technology  Readiness  Assessment 

3.1  Process  Description 

3.2  Critical  Technologies 

3.3  Assessment  of  Maturity 

3.3.1  First  CTE  or  Category  of  Technology 

3.3.2  Next  CTE  or  Category  of  Technology 


3.4  Summary  of  TRLs  by  Technology 
^  4.0  Conclusion 


A  TRA  is  a  technical  report  with  references 
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TRA  Technical  Report  Section  3.1 
Should  Identify  IRT  Members 


Operational  Planning  System  Example 

•  The  TRA  IRT  is  composed  of  system  engineers  from  an  FFRDC  and 
Goodness  University.  None  of  their  employers  have  a  contract  with 
the  program  office.  The  four  principals  leading  the  assessment  are 

Mr.  X,  FFRDC  Corporation.  Mr.  X  has  26  years  of  computer  systems  expe¬ 
rience  and  16  years  of  experience  in  the  environment.  He  holds  a  Bachelor 
of  Arts  in  Quantitative  Methods  from  the  University  of  X  and  a  Master  of 
Science  in  Information  Systems  from  the  School  of  Engineering  from  the 
University  of  X 

Mr.  Y,  FFRDC  Corporation.  Mr.  Y  has  24  years  of  computer  systems  expe¬ 
rience  and  9  years  of  experience  in  the  environment.  He  holds  a  Bachelor 
of  Science  in  Computer  Science  from  the  University  of  Y  and  a  Master  of 
Science  in  Management  Information  Systems  from  the  University  of  Y 

-  Mr.  Z,  FFRDC  Corporation.  Mr.  Z  has  7  years  of  computer  systems 
experience.  He  holds  a  Bachelor  and  Master  of  Science  in  Electrical 
Engineering  and  Computer  Science  from  the  Institute  of  Z 

Dr.  A,  Goodness  University,  Senior  Technology  Consultant,  College  of 
Information  Science  and  Technology 
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TRA  Technical  Report  Section  3.3 
Assessment  of  Maturity 


•  Include  all  the  IRT’s  deliberations,  findings,  and 
conclusions 

•  Present  the  evidence  and  rationale  for  the  final  assess¬ 
ment  clearly  and  logically 

-  Explain  how  the  material  was  used  or  interpreted  to  make 
the  assessment 

•  Evidence  could  include  records  of  tests/applications  of  the 
technology,  technical  papers,  reports,  presentations,  and  so 
forth 

-  Reference  the  sources  and  the  pages  in  these  sources  for 
the  evidence  presented  for  determining  the  TRL 

•  Vague  references  to  test  results  or  test  documents  are  not 
sufficient 

•  The  report  should  rely  upon  a  formally  identified  body  of 
evidence,  which  must  be  either  included  or  accessible 
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Component  S&T  Executive  Contributions 


•  Provide  any  other  pertinent  technical  information  in  the 

TRA  technical  report 

-  Material  provided  by  the  S&T  Executive  should  be  clearly 
differentiated  from  the  material  provided  by  the  IRT 

•  TRA  report  cover  letter 

-  Indicate  agreements  or  disagreements  with  the  findings  of 
the  IRT 

-  A  PM’s  TRL  assessments  can  also  be  included  in  the 
cover  letter 
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TRAs  Should  Not  Include  Recommendations 
for  a  Particular  Programmatic  Decision 


•  Example  1 

-  “...as  part  of  a  risk  mitigation  plan,  eight  prototype  vehicles 
with  the  system  are  undergoing  testing.  Based  on  the 
results  to  date,  the  system  is  considered  mature  enough  to 
enter  low  rate  production.” 

•  Example  2 

-  “...  the  maturity  of  the  critical  technologies,  along  with  the 
associated  risk  mitigation  approaches,  support  entry  into 
System  Development  and  Demonstration  (SDD)  ...” 
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TRAs  Should  Not  Include  Recommendations  for  a 
Particular  Programmatic  Decision  (Continued) 


•  Example  3 

-  “The  Example  3  Product  Manager  identified  two  critical 
technologies  for  the  readiness  of  the  program  to  enter  the 
design  and  development  contract.  The  opinion  of  the 
Example  3  Product  Office  is  that  these  two  critical 
technologies  have  matured  to  a  TRL  level  sufficient  for 
entry  into  the  SDD  contract.  These  two  technologies  will 
have  matured  to  a  TRL  level  sufficient  to  enter  low  rate 
initial  production  (LRIP)  far  ahead  of  schedule” 

-  “Evolution  of  a  system’s  transmit/receive  (T/R)  module 
offers  the  best  available  alternative  for  this  example  to 
meet  T/R  module  requirements  in  SDD.  This  effort  is 
imperative  to  the  success  of  SDD  and  is  true  ‘risk 
reduction.  ’  ” 
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CAE  TRA  Coordination 


•  CAE  approval  is  an  endorsement  of  the  CTE  list  and  the  assessed 

TRLs  only 

•  Maturity  requirements 

-  Subsystem  demonstrated  in  relevant  environment  (TRL  6)  for 
Milestone  B 

-  Prototype  demonstrated  in  an  operational  environment  (TRL  7)  for 
Milestone  C 

•  Three  options  if  a  technology  is  not  mature 

-  Request  a  delay  for  the  Milestone  review  until  all  CTEs  are  at  the  requisite 
maturity  level 

-  Use  alternative,  mature  technologies 

-  As  a  last  resort,  carry  immature  technologies  into  the  Milestone  review 
and  prepare  a  waiver  (based  on  inability  to  meet  national  security 
objectives)  that  the  MDA  can  submit  to  Congress 


Acquisition  Executive  submits  the  TRA  to  DRD 
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DRD  TRA  Review 


•  Results  of  initial  review 

-  Concur 

-  Request  revisions 

•  If  revised,  results  of  final  review 

-  Concur 

-  Concur  with  reservations 

-  Non-concur 

-  Perform  ITA 


DRD  informs  MDA  of  the  results 
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Outline 


*  Executive  Summary 

*  Introduction 

*  Technology  Maturation 

*  Identifying  Critical  Technology  Elements  (CTEs) 

-  CTE  Examples 

*  Assessing  CTE  Readiness 

-  CTE  Readiness  Examples 

*  The  TRA  Report 

*  Summary 

*  Hyperlinks 

*  Backup 


PM  Roles  and  Responsibilities 


•  Plans  and  funds  the  program’s  risk  reduction  activities  to  ensure  that  CTEs 
reach  the  appropriate  maturity  levels 

•  Informs  the  Component  S&T  Executive  of  the  need  to  conduct  a  TRA 

•  Funds  the  TRA  evaluation  for  his  program 

•  Designates  a  responsible  individual  in  the  program  office  to  organize  all  TRA 
activities 

•  Prepares  a  draft  TRA  schedule  and  incorporates  the  approved  version  in  the 
program’s  IMP  and  IMS 

•  Suggests  to  the  Component  S&T  Executive  the  subject  matter  expertise 
needed  to  perform  the  TRA 

•  Ensures  that  the  IRT  is  familiar  with  the  program 

•  Identifies  possible  CTEs  for  IRT  consideration 

•  Provides  evidence  of  CTE  maturity  to  the  IRT  for  its  assessment,  including 
contractor  data 

•  Provides  technical  expertise  to  the  IRT  as  needed 

•  Drafts  the  section  of  the  TRA  report  containing  a  brief  description  of  the 
program  (program/system  overview,  objectives,  and  descriptions) 
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Component  S&T  Executive 
Roles  and  Responsibilities 


•  Directs  the  conduct  of  the  TRA 

•  Coordinates  on  the  TRA  schedule 

•  Nominates  SMEs  to  be  on  the  IRT 

•  Provides  the  DRD  the  credentials  of  all  prospective  IRT  members 
and  sufficient  information  to  confirm  their  independence  from  the 
program 

•  Trains  IRT  members  on  the  TRA  process 

•  Reviews  the  TRA  report  and  prepares  the  TRA  report  cover  memo¬ 
randum,  which  may  include  additional  technical  information  deemed 
appropriate  to  support  or  disagree  with  IRT  findings 

•  Sends  the  completed  TRA  to  the  CAE  for  official  transmittal  to  the 
DRD  and  furnishes  an  advance  copy  to  the  DRD 

•  Maintains  continuity  in  the  IRT  membership  for  all  TRAs  conducted 
over  the  life  of  a  program,  to  the  maximum  extent  possible 
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IRT  Roles  and  Responsibilities 


•  Keeps  the  Component  S&T  Executive  and  the  DRD  informed 
on  progress  throughout  the  entire  TRA  process 

•  Develops  a  list  of  CTE  candidates  in  conjunction  with  the  PM 

•  Assesses  the  TRLs  for  all  CTEs 

•  Prepares  (or  oversees  the  preparation  of)  elements  of  the 
TRA  report  including  (1)  the  IRT  credentials  and  (2)  IRT 
deliberations,  findings,  conclusions,  and  supporting 
evidence 

-  The  assessment  process  should  not  be  constrained  to  a 
validation  of  a  “program-developed”  position  on  the  TRL 


122 


DRD  Roles  and  Responsibilities 


•  Concurs  with  the  TRA  schedule 

•  Concurs  with  the  composition  of  the  IRT 

•  Reviews  the  candidate  CTE  list  and  identifies  any  changes  necessary  to 
form  the  final  CTE  list.  Additions  to  the  list  can  include  any  special- 
interest  technologies  that  warrant  the  rigor  of  the  formal  TRA  process 

•  Exercises  oversight  by  monitoring  and  evaluating  the  TRA  process  and 
reviewing  the  TRA.  On  the  basis  of  that  review,  a  TRA  revision  may  be 
requested  or  the  DRD  may  conduct  its  own  Independent  Technical 
Assessment  (ITA) 

•  Sends  the  results  of  its  TRA  review  to  the  appropriate  Overarching  Inte¬ 
grated  Product  Team  (OIPT)  and/or  the  Defense  Acquisition  Board  (DAB) 

•  Provides  the  DDR&E  recommendations  concerning  certification 

•  Recommends  technology  maturity  language  for  an  Acquisition  Deci¬ 
sion  Memorandum  (ADM),  noting,  in  particular,  conditions  under  which 
the  new  technology  can  be  inserted  into  the  program 
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Basis  of  Technology  Maturity 
Assessments  Throughout  Acquisition 


Milestone  A 

Milestone  B 

Milestone  C 

Basis  of  CTE 
Identification 

Early  evaluation  of 
technology  maturity 

Current  level  of 
design  and  Capa¬ 
bilities  Develop¬ 
ment  Document 
(CDD)  requirements 

Planned  LRIP 
article  (or  limited 
deployment  version 
of  an  IT  system), 
prior  TRAs,  and 
final  design 

CTE  Identification 
Status 

Potential  CTEs 

CTEs  -  actual 
technologies  in  a 
preliminary  design 

CTEs  of  planned 

LRIP  articles  (or 
limited  deployment 
version  of  an  IT 
system) 

Assessment 

Method 

Evaluated  in  early 
evaluations  of  tech¬ 
nology  maturity  and 
Technology  Matura¬ 
tion  Plans  (TMPs) 

Assessed  in 
Milestone  B  TRA 

Assessed  in 
Milestone  C  TRA 

Documentation 

Informal  submission 
to  DRD  and  corres¬ 
ponding  updates  to 
TDS  appendix 

Milestone  B  TRA 

Milestone  C  TRA 
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References  and  Resources 


•  Defense  Acquisition  Resource  Center 

(https://dap.dau.mil/policv/Paqes/overview.aspx) 

DoD  Directive  (DoDD)  5000. 01, The  Defense  Acquisition  System,  certified 
current  as  of  November  20,  2007 

-  DoD  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition 
System,  dated  December  8,  2008 

-  Defense  Acquisition  Guidebook 

•  Defense  Acquisition  University  (DAU )Continuous  Learning  Module 
CLE021  (https://learn.dau.mil/html/clc/Clc.isp) 

•  Technology  Readiness  Assessment  (TRA)  Deskbook 

(http://www.dod.mil/ddre/doc/DoD  TRA  July  2009  Read  Version.pdf) 

•  Institute  for  Defense  Analyses  (IDA) 

-  Dr.  Dave  Sparrow  (dsparrow@ida.org) 

-  Dr.  Jay  Mandelbaum  (imandelb@ida.org) 

-  Dr.  Michael  May  (mmav@ida.org) 
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Sample  Questions  To  Help  Identify  CTEs 
for  an  Aircraft  (Aerodynamic  Configuration) 


•  Does  the  design  incorporate  a  configuration  that  has 
not  been  used  in  flight? 

•  How  similar  is  the  configuration  to  that  of  aircraft  that 
are  successful? 

•  Does  the  configuration  impose  limitations  on  control 
authority,  stability,  structural  rigidity,  or  strength? 

•  Is  stability  acceptable  at  high  angles  of  attack? 

•  Is  stability  and  control  acceptable  during  configuration 
changes  in  flight? 


Return 
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Sample  Questions  To  Help  Identify  CTEs 
for  a  Networked  Communication  System 


•  Do  the  requirements  for  throughput,  data  latency, 
security  or  reliability  imply  that  a  new  or  novel 
technology  is  required? 

•  Have  the  network  routers  been  used  before  within  the 
required  performance  envelope? 

•  Are  new  or  novel  media  access  control,  coding,  or 
routing  algorithms  needed? 

•  Is  the  multiplexing  schema  new? 

•  Is  the  topology  (logical  and  hardware)  new? 

•  Do  the  peak  and  average  data  rates  require  new  hardware 
or  algorithms  in  the  system? 


Return 
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Attainment  of  Technology  Readiness 

for  Hardware  CTEs 


Accomplishment 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Discovery  of  physical  or  mathematical  principle 

X 

Characterization  of  the  principle 

X 

Application  envisioned  and  described 

X 

Concept  of  application  analyzed 

X 

Critical  functionality  empirically  confirmed 

X 

Proof  of  concept  demonstrated  in  laboratory 

X 

Scale-up  or  other  extension  as  needed  by  concept 

X 

X 

Breadboard  or  component  tested  in  laboratory 

X 

Producibility  and  cost  estimated 

X 

X 

Engineering  Development  Model  (EDM)  of  component  tested  in 
laboratory 

X 

EDM  of  component  tested  in  relevant  environment 

X 

Prototype  component  integrated  into  a  system  EDM 

X 

X 

System  EDM  tested  in  simulated  environment 

X 

System  tested  in  limited  field  experiments 

X 

X 

System  tested  in  relevant  environment 

X 

System  tested  in  operational  environment 

X 

Production  system  tested  in  operational  environment 

X 

Production  system  proven  in  mission  operations 

X 

Return 
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Note:  This  is  not  a  comprehensive  checklist. 

It  only  provides  examples  of  supporting  knowledge-based  events. 


Attainment  of  Technology  Readiness 

for  Software  CTEs 


Accomplishment 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Discovery  of  mathematical  principle  or  algorithm 

X 

Characterization  of  the  principle 

X 

Application  envisioned  and  described 

X 

Concept  of  application  analyzed 

X 

Critical  functionality  empirically  confirmed  and  implemented 
software 

X 

Proof  of  concept  demonstrated  in  simulation 

X 

Scale-up  or  other  extension  as  needed  by  concept 

X 

X 

Component  tested  in  simulation 

X 

Producibility  and  cost  estimated 

X 

X 

Software  component  tested  in  an  integration  laboratory 

X 

Software  component  tested  in  a  relevant  environment 

X 

Prototype  component  integrated  into  a  system  prototype 

X 

X 

System  tested  in  a  simulated  environment 

X 

System  tested  in  a  limited  field  experiments 

X 

X 

System  tested  in  a  relevant  environment 

X 

System  tested  in  an  operational  environment 

X 

Production  system  tested  in  an  operational  environment 

X 

Production  system  proven  in  mission  operations 

X 

Return 
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Note:  This  is  not  a  comprehensive  checklist. 

It  only  provides  examples  of  supporting  knowledge-based  events. 


IT  TRA  Challenges 


•  TRA/TRL  model  derived  for  hardware-oriented  systems 

•  Increasing  number  of  defense  acquisitions  are  software 
intensive 

-  Few  hardware  or  software  elements  can  be  singled  out  as 
CTEs 

•  New  IT  issues  include 

-  Interfaces 

-  Throughput 

-  Scalability 

-  External  dependencies 

-  Process  reengineering 

-  Information  assurance 

•  Environment/architecture  plays  a  greater  role 


Return 
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Software-Intensive  Systems 
Fall  Into  Five  Broad  Areas 


•  Information  Systems  and  Business  Systems 

•  Networked  Communications  Systems 

•  Mission  Planning  Systems 

•  Embedded  IT  in  Tactical  Systems 
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Return 


Information  Systems  and  Business  Systems 


•  Challenges 

-  Large  COTS  applications 

-  Integration  with  legacy 
business  applications 

-  Integration  in  final 
environment 

-  Data  management 

-  End-to-end 
responsiveness 

-  Scalability 


Recommendations 

-  Start  with  lists  of  COTS 
products 

•  Focus  on  critical  applications 
used  in  a  new  or  novel  way 

•  Use  pilot  experience  to  justify 
TRLs  of  6  and  above 

-  Include  integrating  technologies 
where  applicable 

-  Pay  attention  to  DoD-unique 
environments 

-  Address  system-level  issues 

•  Responsiveness,  scalability,  and 
so  forth 
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Networked  Communications  Systems 


•  Challenges 

-  Services’  focus 

-  Consolidation  of  user 
needs  and  anticipated 
growth 

-  Managing  standards 

-  Technology  rollover 

•  Ability  to  provide  services 

•  May  transcend  individual 
products 

•  Information  assurance 


Recommendations 

-  Start  with  technologies  that 
enable  one  or  more  services 

-  Avoid  process  issues  except 
where  enabled  by  technology 

•  Roll-out,  configuration 
management 

-  Establish  Technology  Transition 
Agreements  (TTAs)  where  DoD 
needs  are  not  met  by 
commercial  technologies  (e.g., 
mobile  ad-hoc  network 
protocols) 

-  Consider  market  capabilities  as 
well  as  specific  technologies 
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Mission  Planning  Systems 


•  Challenges 

-  Reliance  on  external  data 
sources 

-  Mixed  COTS/GOTS 
components 

-  Infrastructure  upkeep  and 
modernization 

-  Technology  turnover 

-  Scalability  and 
responsiveness 


Recommendations 

-  Start  with  required  functionality 
and  supporting  technologies 

-  Identify  critical  data  dependencies 
on  external  programs 

-  Assess  ability  to  succeed  based 
upon  total  suite  of  data  suppliers/ 
users  and  infrastructure,  not  just 
application  maturity 
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Embedded  IT  in  Tactical  Systems 


Challenges 

-  Lots  of  developed  software 

-  Military-unique  environments 

•  Radiation  hardened, 
shock/vibration,  high 
reliability 

-  Military-unique  functionality 

•  IT  as  an  enabler 


Recommendations 

-  Start  with  function  domains 
or  WBS 

-  IT  typically  not  a  CTE  except 
where  consolidated  com¬ 
puting  requirements  are  used 

-  For  COTS,  carefully  examine 
relevant  and  operational 
environment  success  when 
rating  technology  readiness 

-  Do  not  address  developer 
capabilities  in  assessing 
technology  maturity 
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How  TRAs  Got  Started 


•  “Program  managers’  ability  to  reject  immature  technologies  is  hampered  by 

(1)  untradable  requirements  that  force  acceptance  of  technologies  despite  their 
immaturity  and  (2)  reliance  on  tools  that  fail  to  alert  the  managers  of  the  high  risks 
that  would  prompt  such  a  rejection.”  (GAO/NSIAD-99-162) 

•  “Identify  each  case  in  which  a  major  defense  acquisition  program  entered  system 
development  and  demonstration  ...  into  which  key  technology  has  been  incorpo¬ 
rated  that  does  not  meet  the  technology  maturity  requirement ...  and  provide  a 
justification  for  why  such  key  technology  was  incorporated  and  identify  any  deter¬ 
mination  of  technological  maturity  with  which  the  Deputy  Under  Secretary  of 
Defense  for  Science  and  Technology  did  not  concur  and  explain  how  the  issue  has 
been  resolved.”  (National  Defense  Authorization  Act  for  Fiscal  Year  2002) 

•  “The  management  and  mitigation  of  technology  risk,  which  allows  less  costly  and 
less  time-consuming  systems  development,  are  crucial  parts  of  overall  program 
management  and  is  especially  relevant  to  meeting  cost  and  schedule  goals.  Objec¬ 
tive  assessment  of  technology  maturity  and  risk  shall  be  a  routine  aspect  of  DoD 
acquisition.”  (DoDI  5000.2,  Enclosure  2,  paragraph  5.d(4)) 


Stop  launching  programs  before  technologies  are  mature 
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Best  Practices  for  a  Preliminary  TRA 
at  Milestone  A  (Only  Applies  to  Ships) 


Example 

•  Use  the  TRA  to  identify  areas  for  management  focus 

-  Create  critical  technology  IPTs 

•  No  contract  award  yet 

-  Update  the  TRA  after  a  selection  decision 

•  No  TRL  requirements 

-  TRL  of  3  or  lower  implies  higher  technology  risk 

-  Technology  Development  Phase  generally  mature  tech¬ 
nology  from  TRL  4  to  TRL  6 

-  UseTTA 


141 


Nature  of  the  TRA  at  Milestone  C 


•  Start  where  Milestone  B  TRA  left  off 

•  Use  the  same  IRT  that  was  used  at  the 
TRA  at  Milestone  B 

•  Based  upon  the  detailed  design  documentation  in  the 
product  baseline 

-  Determine  if  any  new  CTEs  have  emerged 

-  Pay  careful  attention  to  operational  environment  implica¬ 
tions,  especially  for  COTS  products 

-  Assess  maturity 

•  Performance-related  CTEs  should  be  at  least  TRL  7 


Recommended 
Best  Practice 
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PM 

responsibility 
to  provide 


IRT  Information  Needs 


•  Program  overview  to  set  the  foundation  for  the  CTE 
assessments 

-  Concept  of  operations  (CONOPS) 

-  Program  master  schedule 

•  Identify  significant  milestones,  items  on  the  critical  path,  and  status  of 
progress 

-  Operational  performance  requirements 

•  Highlight  KPPs  in  general  and  those  operational  requirements  that  will 
be  directly  influenced  by  the  CTEs  to  be  assessed 

-  The  challenges  associated  with  the  CTEs  to  be  assessed 

-  Technology  maturation  roadmap 

•  Highlight  those  maturation  events  that  have  been  accomplished  and 
those  that  have  yet  to  occur 

-  Overall  system  architecture 

•  Highlight  the  CTE  system/subsystem  elements  that  will  be  assessed 
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PM 

responsibility 
to  provide 


IRT  Information  Needs 
(Continued) 


•  Introduction  to  the  subsystems  containing  the  CTEs 

-  Technical  description  of  the  subsystem,  to  include  physical  architecture, 
highlighting  CTEs  (components  and/or  packaging):  explain  why  other 
technologies  within  subsystem  are  non-critical  and  differentiate 
subsystem  and  elements  from  those  of  potentially  similar  designs  (i.e., 
highlight  any  uniqueness) 

-  Description  of  the  subsystem’s  intended  function  in  the  design 

•  Significance  of  the  CTEs  relative  to  the  subsystem 

•  Significance  of  the  subsystem  relative  to  the  system  overall  design 

•  Traceability  of  the  subsystem  relative  to  the  applicable  operational  require¬ 
ments.  State  whether  subsystem  will  impact  a  KPP 

-  Schedule  for  the  design  and  integration  of  the  subsystem,  clearly 
identifying  critical  path  events  and,  if  relevant,  expectation/deliveries 
from  suppliers 

-  Block  diagram  and  risk  assessment  for  the  subsystem 

-  Roadmap  of  ongoing  and  planned  maturation  activities  and  how  these 
events  can  influence  the  master  design  schedule 
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PM 

responsibility 
to  provide 


IRT  Information  Needs 
(Continued) 


•  Status  of  CTE 

-  Accomplishments  that  directly  reflect  CTE  maturation 

•  Use  TRL  rating  factors  as  a  guideline 

•  State  quantitative  facts  where  possible  to  temper  and  legitimize  the  significance 
of  the  technology  maturation  accomplishments 

•  Describe  the  measurement  environment  and  methodology  used 

•  Identify  who  witnessed  the  subsystem/technology  maturation  accomplishments 

-  Tangible  evidence  of  CTE  maturation  accomplishments  (e.g.,  hardware, 
pictures,  displays,  technical  papers,  reports,  and  so  forth) 

•  Clearly  state  what  is  and  is  not  represented  by  the  evidence 

-  Relevant  CTE  maturation  leveraged  from  other  programs 

•  Clearly  state  any  differences  between  this  program  and  the  leveraged  program, 
to  appreciate  significance  of  maturation  events 

-  Significant  maturation  events  that  fall  short  or  have  not  been 
accomplished 
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Preconditions  for 

Entering  EMD  With  Immature  CTEs 


•  MDA  submits  waiver  to  Congress  describing  why 
waiving  these  requirements  was  necessary  to  meet 
national  security  objectives 

•  A  sound  technical  basis  exists  for  expecting  the 
immature  technology  to  prove  adequate  after  a 
demonstration 

•  If  the  demonstration  is  unsuccessful,  a  substitute  mature 
technology  is  available  and  can  be  used 

•  The  program  plan  can  accommodate  use  of  either  tech¬ 
nology  from  funding,  performance,  and  schedule 
perspectives 
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Use  of  TRA  and  TRL  Terminology 
Acquisition  Community 


•  TRAs  help  ensure  that  the  technology  being  used  in  acquisi¬ 
tion  programs  is  mature 

-  Use  of  immature  technologies  leads  to  cost  growth  and  schedule 
slippage 

•  TRAs  provide  the  basis  for  DDR&E  to  advise  the  MDA  on 
10  U.S.C.  2366b  certification  for  technology  maturity  at 
Milestone  B/KDP  B 

•  TRLs  are  the  maturity  metric  for  CTEs  in  TRAs.  TRLs  5-9  are 
applicable 

-  Environments  and  the  performance  requirements  defined  by  a 
program  of  record 

•  TRAs  performed  at  Milestones  B  and  C  and  at  program  initia¬ 
tion  for  ships 
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Use  of  TRA  and  TRL  Terminology 

S&T  Community 


•  TRAs  are  an  acquisition  construct 

-  They  are  not  performed  on  an  S&T  project 

•  TRLs  may  be  used  as  a  maturity  metric  for  technologies 
in  a  technology  development  project.  TRLs  1-6  are 
applicable 

-  TRL  definitions  and  descriptions  successively  spell  out 
progress  (as  measured  by  tests)  toward  a  goal 

•  TRLs  may  be  used  as  part  of  a  technology  managers’ 
ongoing  assessment  of  a  technology  or  technologies 


Use  of  TRLs  5  and  6  overlaps  with  the  acquisition  community 
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Issues  Arising  From  an  Overlap  in  Terminology 


•  Normally,  demonstrating  a  technology  beyond  TRL  4  requires 
more  resources  than  maturing  the  technology  through  TRLs  1-3 

-  Higher  level  assemblies  needed 

-  More  refined  components  are  needed 

-  More  broad  scale  tests  are  needed 

•  Such  resources  often  obtained  from  programs  of  record  as  activi¬ 
ties  shift  from  the  realm  of  technology  advancement  to  technology 
transition  and  insertion 

•  Misunderstanding  of  TRLs  5  and  6  has  led  to  misuse  of  the  termi¬ 
nology  when  competing  for  these  resources 

-  May  damage  the  S&T  program  and/or  the  TRA  process 

-  May  create  the  wrong  impression  with  leadership 


Misuse  of  TRA  and  TRL  terminology  and  concepts 
may  lead  to  negative  unintended  consequences 
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TRL  4  Is  the  Breakpoint 
Between  Invention  and  Application 


•  TRLs  1-3  involve  development  of  functionality,  mostly 
independent  of  the  application 

•  To  achieve  TRL  4 

-  Must  begin  integration  of  components  to  represent  how 
they  would  be  used  in  a  fieldable  application 

-  Must  have  a  generic  application  in  mind  without  knowing 
exactly  how  that  application  will  be  used 
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TRL  4  Is  the  Breakpoint  Between 
Invention  and  Application  (Continued) 


•  To  achieve  TRL  5  or  higher 

-  Must  be  in  the  context  of  an  application  for  a  program  of 
record 

•  The  application  provides  the  both  the  metric  (speed,  energy 
density  ...)  and  the  threshold  (10  m/s,  lOOJ/g  ...) 

-  Must  have  an  understanding  of  the  relevant  environment 

•  The  relevant  environment  cannot  be  determined  without  an 
understanding  of  requirements  and  intended  operational 
use,  as  defined  in  programmatic  documents 
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TRL  4  Is  the  Breakpoint  Between 
Invention  and  Application  (Continued) 


•  Gun  propellant  example 

-  TRL  1 :  Theoretical  studies  and  computer  models  lead  to  synthesis 
and  characterization  of  a  new  energetic  material  for  a  propellant 

-  TRL  2:  New  material  synthesized  and  characterized,  and  potential 
performance  of  propellants  mapped  via  computer  codes 

-  TRL  3:  New  propellants  prepared  at  small  scale,  and  performance, 
processing,  and  physical  properties  characterized 

-  TRL  4:  Based  on  TRL  1-3  data,  propellant  designed  for  a  specific 
application  and  near-full-scale  tests  performed  to  confirm  com¬ 
puter  modeling 

-  TRL  5:  New  propellant  produced  in  quantity  and  evaluated  in  the 
near-final  system  configuration 
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Milestone  B  Requirement:  Demonstration  or 
Validation  in  a  Relevant  Environment  (TRL  6) 


Relevant  Environment:  a  set  of  stressing  conditions  represen¬ 
tative  of  the  full  spectrum  of  relevant  operational  employments, 
which  are  applied  to  a  CTE  as  part  of  a  component  (TRL  5)  or 
system/subsystem  (TRL  6)  model  or  prototype  to  identify  whether 
any  design  changes  or  fixes  are  needed  to  support  the  required 
(threshold)  functionality 
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Demonstration  or  Validation  of  a  Technology 
in  a  Relevant  or  Operational  Environment 


•  Requires  successful  trial  testing  that  either 

-  Shows  that  the  technology  satisfies  functional  need 
across  the  full  spectrum  of  operational  employments  or 

-  Shows  that  the  technology  satisfies  the  functional  need  for 
some  important  (stressing)  operational  employment  and 
that  it  uses  accepted  techniques  to  extend  confidence 
over  all  required  operational  employments 
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S&T  Practices  To  Better  Support  TRAs 
and  the  Acquisition  Process 


•  Not  labeling  the  technology  assessments  performed  by 
the  S&T  community  as  a  TRA 

-  Misuses  the  term  in  a  way  that  misleads  stakeholders 

-  May  damage  both  TRA  and  technology  proponent’s 
reputation 

•  Not  justifying  the  need  for  research  (dollars)  based  on 
achieving  TRL  5/6  without  the  metrics  and  the  threshold 
provided  by  a  program  of  record 
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S&T  Practices  To  Better  Support  TRAs 
and  the  Acquisition  Process  (Continued) 


•  Applying  judgment  when  trying  to  differentiate  the  relevant  environ¬ 
ment  from  the  operational  environment  to  maximize  test  efficiency 

-  Environments  tested  should  be  stressing  enough  to  be  persuasive 
•  Being  exhaustive  is  usually  too  expensive 

Example 

•  Launching  a  satellite  should  not  be  on  the  critical  path  for  design 
and  demonstration 

•  Relevant  environment  depends  on  what  is  stressing 

-  For  example,  thermal  load,  radiation  in  space,  g-forces  during  launch 

-  Can  be  tested  and  demonstrated  in  the  lab 

•  Technical  expertise  ensures  that  the  stressing  portion  of  the  envi¬ 
ronment  is  demonstrated.  No  expensive,  exhaustive  tests  applied  to 
non-critical  elements 
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S&T  Practices  To  Better  Support  TRAs 
and  the  Acquisition  Process  (Continued) 


•  Continuing  promising  technology  development  at  TRL  4  when 
there  is  no  program  of  record 

-  While  TRLs  5/6  are  achieved  with  a  successful  demonstration,  a 
large  number  of  useful  activities  could  take  place  at  TRL  4 

•  Completing  an  extensive  performance  characterization  rather  than  a 
“point  demonstration”  is  helpful 

-  Provides  information  on  how  to  incorporate  the  technology  into  a  design 

-  Enables  more  rapid  insertion 

-  Supports  knowledge-based  acquisition  decisions 

•  A  technology’s  capability  can  be  advanced  using  metrics  of 
interest  without  knowing  the  particular  thresholds 

•  Improvements  can  be  planned  on  the  basis  of  draft 
requirements 


Continuing  development  also  applies  to  TRL  5/6 
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S&T  Practices  To  Better  Support  TRAs 
and  the  Acquisition  Process  (Continued) 


•  Preparing  to  help  programs  of  record  achieve  TRLs  5/6  via  exper¬ 
tise  with  the  technology  and  the  test  design,  as  they  reach  back  to 
the  tech  base  for  solutions 

-  Neither  labs  nor  program  offices  are  organized  or  staffed  to  conduct  the 
realistic  demonstrations  of  highly  integrated  components  needed  to 
mature  technologies  to  TRL  5/6 

-  S&T  personnel  (and  institutions)  should  transition  into  a  supporting  role 

Example 

•  Armor-piercing,  Fin-stabilized  Discarding  Sabot  (APFSDS)  had  nearly 
maxed  performance  capabilities 

•  The  1984  Armament  Enhancement  Initiative  was  established  to 
reduce  sabot  weight  (partnership  between  S&T  and  acquisition) 

•  By  1987,  requirement  had  been  established  for  new  composite  sabot, 
which  was  fielded  in  1992 

•  Cycle  repeated  itself  for  fielding  a  more  advanced  sabot  in  2003 


159 


S&T  Practices  To  Better  Support  TRAs 
and  the  Acquisition  Process  (Continued) 


•  Differentiating  proof-of-principle  demonstration  (TRL  3) 
from  demonstration  in  a  (requirements-defined)  relevant 
environment  (TRL  5/6) 

-  For  TRL  3:  do  not  need  to  have  an  application  in  mind 

«  Acquisition  customer  may  say,  “If  you  make  it  work,  I’ll  use  it” 

-  For  TRL  5:  must  be  an  application,  and  components  must 
be  representative  of  use  in  intended  application 

-  For  TRL  6:  ready  to  turn  it  over  to  a  design  engineer 

Overselling  technology  readiness  damages  the  S&T 
community  credibility  as  much  as  overselling  technology 
performance.  May  lead  to  . . . 
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S&T  Practices  To  Better  Support  TRAs 
and  the  Acquisition  Process  (Continued) 


. . .  acquisition  problems  if  program  initiated  with  immature 
technology 


Example 

•  The  regenerative  liquid  propellant  gun  (RLPG)  ultimately 
became  the  CRUSADER  program 

-  When  program  transitioned  from  S&T,  the  concept  was 
proven 

•  All  technology  issues  were  reasonably  well  recognized 

•  Plan  was  to  solve  the  problems  in  engineering 

-  Eventual  failure  (program  cancellation)  was  associated 
with  the  difficulties  encountered  when  transferring  the 
technology  to  practical  hardware 
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S&T  Practices  To  Better  Support  TRAs 
and  the  Acquisition  Process  (Continued) 


•  Avoiding  the  use  of  TRLs  as  a  sole,  governing  measure  for 
managing  S&T  programs 

-  TRLs  are  a  static  metric  and  represent  snapshot  in  time.  TRLs  do 
not  assess  difficulty  of  advancement 

-  TRLs  lack  high  specificity.  Much  more  information  needs  to  be 
conveyed 

•  Should  lay  out  specific  technical  goals  to  evaluate  technology  status/ 
progress 

-  Could  lead  to  a  premature  stoppage  of  development  efforts  as 
soon  as  next  TRL  is  reached 

•  Using  TRLs  a  high-level  metric  for  managing  a  balanced  port¬ 
folio  of  investments  from  basic  research  to  exploratory  devel¬ 
opment  of  components 

-  Helps  avoid  under  emphasis  on  basic  principles  or  concept  for¬ 
mulation  (TRLs  1  and  2)  in  favor  of  research  on  proof  of  principle 
or  demonstration  in  lab  (TRLs  3  and  4) 
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Acquisition  Practices 
To  Improve  Linkages  With  S&T 


•  Developing  (in  conjunction  with  the  S&T  community)  a 
TMP  to  identify  how  technologies  will  be  demonstrated 
in  a  relevant  environment  by  Milestone  B 

•  Establishing  measurable  technical  performance  require¬ 
ments  as  technology  transition  exit  criteria  to  achieve 
TRL  6  for  CTEs 

-  Fully  describe  the  relevant  environment  in  TTAs 

-  Include  metrics  and  thresholds  in  a  relevant  environment 

-  Do  not  specify  TRL  6  as  an  exit  criterion 
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Acquisition  Practices 

To  Improve  Linkages  With  S&T  (Continued) 


•  Shifting  necessary  resources  (funding  and  personnel)  to 
the  Technology  Development  Phase 

•  Accounting  for  the  event-driven  nature  of  S&T  processes 
when  developing  schedules 

-  Applying  schedule-driven  constraints  may  compromise 
technology  development  and  lead  to  immature  technologies 
at  Milestone  B 

-  Backup  plans  and  alternatives  to  technologies  less  than 
TRL  6  are  advisable 
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